Discrete Math - Chapter 12
.pdf
School
David and Mary Thomson Collegiate Institute**We aren't endorsed by this school
Course
MATH ELEE2110U
Subject
Computer Science
Date
Jan 10, 2025
Pages
20
Uploaded by UltraSeaUrchinMaster867
CHAPTER 12AlgorithmsContents12.1Algorithm Requirements21112.2Algorithmic Paradigms21312.3Complexity of Algorithms21512.4Measuring Algorithm Efficiency21712.5Sorting Algorithms22312.6Search Algorithms227The termalgorithmis the corrupted version of the last name of the famous mathematicianal-Khowarizmi, whose book on Hindu numerals was principally responsible forspreading the Hindu-Arabic numeral system throughout Europe and the developmentof mathematics during the European Renaissance. Algorithms are very widely used ineveryday life in one way or another. For instance, there are several search engines onthe Internet, where each has its own proprietary algorithm that ranks websites for eachkeyword or combination of keywords. From seeking the shortest route and online shop-ping to biometric authentication and video compression, algorithms are everywhere inmodern life. This chapter briefly discusses some aspects of algorithms and describes severalsorting and search methods.12.1 Algorithm RequirementsAnalgorithmis afinite unambiguous sequence of instructions, set of rules, or number ofsteps that involves repetition of an operation or reiteration of a procedure for performinga computation, solving a mathematical problem, or accomplishing some end in afiniteamount of time. Some of the important requirements for an algorithm are as follows:•An algorithm has input values from a specified set.•An algorithm produces the output values from the input values, which is the solutionto a problem.•An algorithm possessesfiniteness, that is, it produces the output after afinite numberof steps.•An algorithm possesses definiteness, that is, all steps of the algorithm are preciselydefined using unambiguous, well-defined operations.•An algorithm possesses correctness, that is, it produces correct output values for anyset of input values.Discrete MathematicsISBN 978-0-12-820656-0,https://doi.org/10.1016/B978-0-12-820656-0.00012-5©2023 Elsevier Inc.All rights reserved.211
•An algorithm possesses effectiveness, that is, it performs each step precisely and in afinite amount of time, and no step can be impossible to do, such as division by zero.•An algorithm is well-ordered, as a computer can only execute an algorithm if it knowsthe exact order of steps to perform.•An algorithm possesses generality, that is, it should accept any general set of inputvalues.Example 12.1Noting a chessboard (a square board divided into 64 alternating dark and lightsquares) is referred to as an 8±8 board, the total number of squares of all sizes(from the largest square to the smallest squares) in ann±nchessboard is as follows:S¼Xni¼1i2Describe the steps of an algorithm thatfinds the sumSfor an integern²1.SolutionNoting that this algorithm meets all requirements, the following sequence of stepsis required:(1)Inputn.(2)SetS¼0.(3)Seti¼1.(4)Computei2¼i±i.(5)Addi2toS:(6)Add 1 toi.(7)Compareiton; ifi>n, then outputSand stop.(8)Go back to step (4).An algorithm is called optimal for the solution of a problem with respect to a specificoperation (e.g., the number of comparisons) if there is no algorithm for solving this prob-lem using a fewer number of operations.In order to implement an algorithm using a computer, aflowchart or a pseudocodemay be required. Aflowchartis a graphical method of presenting an algorithm to illustrateitsflow of execution. Aflowchart shows how theflow of execution proceeds from onestatement to the next until the end of the algorithm is reached. Apseudocodeis a semi-formal language, an intermediate step between a text description of an algorithm and animplementation of the algorithm in a programming language. There are no required syn-tax rules for pseudocode, as it is intended for human reading rather than machine reading.However, there are many useful notational conventions to eliminate ambiguity and aid in212Discrete Mathematics
clarity. Theflowchart or pseudocode of an algorithm can be converted into a program-ming language of choice, such as Java or Cþþ.There are some problems for which it can be shown that no algorithm exists for solv-ing them, one such problem is the halting problem. Thehalting problemis a procedurethat takes as input a computer program and the input to the program and determineswhether the program will eventuallyfinish running (i.e., will ultimately stop) or continueto run forever. It is obviously very important to have such a procedure to test whether aprogram may enter into an infinite loop when writing and debugging programs. AlanTuring proved that a general algorithm to solve the halting problem for all possibleprogram-input pairs cannot exist. Based on proof by contradiction, it can be shownthat the halting problem on Turing machines (an abstract computational model that per-forms computations by reading and writing to an unlimited amount of memory) is un-decidable, hence unsolvable, as the halting problem seeks an algorithm that works for allprograms and all input data.12.2 Algorithmic ParadigmsAnalgorithmic paradigmis an abstraction higher than the notion of an algorithm. Analgorithmic paradigm is a generic model, based on a particular approach, which underliesthe design of a class of algorithms for solving a multitude of problems. Some of theimportant algorithmic paradigms are as follows:•Brute-force algorithms: The brute-force algorithm is a simple algorithm in conceptthat blindly iterates all possible solutions to search for one or more than one solutionthat may solve a problem without any regard to the heavy computational require-ments. It takes an inefficient approach for solving problems, as it does not take advan-tage of the special structure of the problem. Examples include using all possiblepermutations of numbers to open a safe,finding the largest number in a list ofnumbers, sorting problems, such as the bubble, insertion, and selection sorts, and poll-ing a multitude of communication devices to determine those having messages totransmit.•Divide-and-conquer algorithms: The divide-and-conquer algorithm is an effectivealgorithm that works by recursively breaking down a problem into two or more sub-problems of the same or related type until these become simple enough to be solveddirectly and rather easily. The solutions to the subproblems are then combined to givea solution to the original problem. Examples include sorting problems, such as quick-sort and merge sort, binary search, solving the closest pair problem, routing mail bysorting letters into separate bags for different geographical areas, and applying the lawof total probability.Algorithms213
•Backtracking algorithms: The backtracking algorithm incrementally builds candidates tothe solution and abandons a candidate (i.e., backtracks) as soon as it determines that thecandidate cannot possibly be a part of a valid solution. It is generally applied tofind so-lutions to some constrained optimization problems. Examples include the eight queenspuzzle, which asks for all arrangements of eight chess queens so that no queen attacksany other, crosswords, Sudoku, the knapsack problem, andfinding spanning trees.•Dynamic programming: The dynamic programming algorithm can be effectively usedfor solving a complex problem by recursively breaking down the problem. It requiresthat overlapping subproblems exist, and the optimal solution of the problem can beobtained using optimal solutions of its subproblems stored in memory with the helpof a recurrence relation. Examples include the scheduling problem, Fibonaccinumbers, matrix chain multiplication, and the traveling salesman problem.•Probabilistic algorithms: The probabilistic algorithm can solve problems that cannotbe easily solved by deterministic algorithms. In contrast to a deterministic algorithm,which always follows the same steps for a given input and has to go through a verylarge number of possible cases, the probabilistic algorithm makes some randomchoices at some steps, which may lead to different outputs in much fewer steps,but with a tiny probability that thefinal answer may not be correct. Examples includeMonte Carlo algorithm in quality control, collisions in hashing functions, andBayesian spamfilters.•Greedy algorithms: The greedy algorithm is one of the simplest and most intuitive al-gorithms that is used in optimization problems, and it often leads to an optimal solu-tion. The algorithm makes the optimal choice at each step as it attempts tofind theminimum or maximum value of some parameter. Greedy algorithms sometimes failto produce optimal solutions because they do not consider all the data, as the choicemade by a greedy algorithm may depend on choices it has made so far, but it is notaware of future choices it could make. A greedy algorithm in a shortsighted manneridentifies an optimal subproblem in the problem. With the goal of minimizing ormaximizing the parameter of interest, create an iterative way to go through all ofthe subproblems so as to build a solution. Examples may includefinding the shortestpath through a graph and a Huffman code used in lossless digital compression of data.Example 12.2Use the greedy algorithm tofind an optimal solution in each of the following twocases:(a)Use the least number of coins to make 92 cents change with quarters, dimes,nickels, and pennies.(b)Determine the path with the largest sum inFig. 12.1.214Discrete Mathematics
Solution(a)In this case, the greedy algorithm produces an optimal solution. To makechange for 92 cents for the least number of coins, the focus at any step ison the largest possible coin, that is, on quarters, dimes, nickels, and pennies,respectively. First, three quarters are selected, leaving 17 cents; next, one dimeis selected, leaving 7 cents; next a nickel is selected, leaving 2 cents; andfinally2 pennies. The total number of coins is thus 7ð¼3þ1þ1þ2Þ, as92¼ ð3±25Þ þ ð1±10Þ þ ð1±5Þ þ ð2±1Þ.(b)In this case, the greedy algorithm does not produce an optimal solution. Thegreedy algorithm fails tofind the largest sum simply because it makes decisionsbased only on the information it has at any one step, without regard to theoverall problem. In order to reach the largest sum, at each step, the greedyalgorithm chooses what appears to be the optimal immediate choice. It there-fore chooses 30 instead of 20 at the second step and thus does not reach thebest solution, which is 100ð¼10þ20þ70Þ, and it mistakenlyfinds90ð¼10þ30þ50Þas the largest sum.12.3 Complexity of AlgorithmsThe two main measures for the computational complexity of an algorithm are as follows:•Space complexity: It is measured by the maximum amount of computer memoryneeded in the execution of the algorithm, and the requirement is frequently a mul-tiple of the data size.•Time complexity: It is measured by counting the number of key operations using thesize of the input as its argument.The termcomplexitygenerally refers to the running time of the algorithm. The func-tionfðnÞ, representing the time complexity of an algorithm, is measured by the followingtwo factors:(i)The sizenof the input data and the characteristics of the particular input data.(ii)The number of basic key operations that must be performed when the algorithm isexecuted, while noting they generally include addition, subtraction, multiplication,division, and comparison.Fig. 12.1Possible paths for Example 12.2.Algorithms215
Example 12.3Discuss the algorithms required to perform the following mathematical operations:(a)Matrix multiplication using direct method.(b)Polynomial evaluation using Horner’s method.Solution(a)In the multiplication of two matrices, order matters (i.e., matrix multipli-cation is not commutative). In order to multiply two matrices, we mustfirst make sure that the number of columns in thefirst matrix equalsthe number of rows in the second matrix, as this is the prerequisite formultiplication. The widely known direct method is as follows: if thefirstmatrix is anm±nmatrix and the second matrix is ann±pmatrix, wherem, n,andpare all positive integers, then their matrix product is them±pmatrix, whose entries are given by the dot product of the correspondingrow of thefirst matrix and the corresponding column of the second matrix.In other words, we multiply the elements of rowiof thefirst matrix by theelements of columnjin the second matrix to obtain the element of theithrow andjth column of the product. The algorithm to perform this matrixmultiplication thus requiresmpnmultiplications andmpðn³1Þadditions.Note that there are other algorithms that require a smaller number of mul-tiplications and additions to multiply two matrices than the direct methoddoes.(b)Consider the following functionfðxÞ, which is a polynomial of degreen:fðxÞ ¼anxnþan³1xn³1þ:::þa2x2þa1xþa0:Suppose we want to evaluatefðxÞforx¼c, that is,fðcÞ. We can considertwo different algorithms for a polynomial evaluation. Thefirst is known asthe direct method, which requiresnþ ðn³1Þ þ ðn³2Þ þ.þ1¼nðnþ1Þ2multiplications andnadditions. The second is known asHorner’s method,which is based on rewriting the polynomial by successively factoring outx,and is as follows:fðxÞ ¼a0þa1xþa2x2þ:::þan³1xn³1þanxn¼a0þxða1þxða2þ:::þxðan³1þxanÞÞÞ:Horner’s method is significantly more efficient than the direct method, as itrequires onlynmultiplications andnadditions.216Discrete Mathematics
Noting the functionfðnÞrepresents the time complexity of an algorithm for the inputdata of sizen,there are typically two types of time complexity to analyze:(i)Theaverage-case time complexity, that is, the expected value offðnÞ, is usually diffi-cult to analyze, and it is generally assumed that the distribution of the possible inputsis uniform when the actual distribution is unknown, and the uniform distributionmay not actually apply to real situations.(ii)Theworst-case time complexity, that is, the maximum value offðnÞ, is easier than theaverage-case complexity to analyze, as this complexity is based on the largest numberof operations required, as a solution for any possible input is guaranteed.12.4 Measuring Algorithm EfficiencyAlgorithmic efficiencyis a property of an algorithm that relates to the amount of compu-tational resources used by the algorithm. An algorithm must be analyzed to determine itsresource usage, and the efficiency of an algorithm can be measured based on the usage ofdifferent resources. The asymptotic growth of functions is commonly used in the analysisof algorithms to estimate the run time and the amount of memory they require. It is veryimportant to provide approximations that make it easy to estimate the large-scale differ-ences in algorithmic efficiency while ignoring differences of a constant factor and differ-ences that occur only for small sets of input data.The estimates of resources required by an algorithm are represented by the big-Oh(O), big-Omega (U), and big-Theta (Q) notations without being concerned about con-stant multipliers or smaller order terms. Note that the three notationsfðxÞ ¼OðgðxÞÞ,fðxÞ ¼UðgðxÞÞ, andfðxÞ ¼QðgðxÞÞall stand for collections of functions. Hence theequality sign does not mean equality of functions. Moreover, as we are dealing with func-tions representing complexity, these functions take on only positive values. Therefore allreferences to absolute values can be dropped for such functions.The growth of a function representing the complexity of an algorithm can be esti-mated using the big-Onotation as its input grows. LetfðxÞandgðxÞbe real-valuedfunctions defined on the same set of nonnegative real numbers. ThenfðxÞisOðgðxÞÞ, which is read asfðxÞisbig-OhofgðxÞ, if there are real constantsCandksuch thatjfðxÞj ´CjgðxÞjwheneverx>k. This definition indicates thatfðx) growsslower than somefixed multiple ofgðxÞasxgrows slowly without bound. However,the rate of growth of multiple ofgðxÞshould be preferably close to the rate of growthoffðxÞ, wheregðxÞprovides an upper bound for the size offðxÞfor large values ofx.The constantsCandkare calledwitnessesto the relationshipfðxÞisOðgðxÞÞand are notunique. Although there are infinitely many pairs of witnesses, we need only one pair ofwitnesses. The approach tofind a pair of witnesses is tofirstfind a value ofkfor whichthe size ofjfðxÞjcan be easily estimated whenx>k,and thenfind a value ofCforAlgorithms217
whichjfðxÞj ´CjgðxÞjwhenx>k. In short, if for a sufficiently large value ofx, thevalues offðxÞare less than those of a multiple ofgðxÞ, thenfðxÞisOðgðxÞÞ.The following properties of the big-Oestimate are of importance:•When we havefðxÞ ¼OðgðxÞÞandjhðxÞj>jgðxÞj, we then havefðxÞ ¼OðhðxÞÞ:•WhenfðxÞis a polynomial of degreen, we then havefðxÞ ¼OðxnÞ.•If we havef1ðxÞ ¼Oðg1ðxÞÞandf2ðxÞ ¼Oðg2ðxÞÞ, then we havef1ðxÞ þf2ðxÞ ¼Oðmaxðjg1ðxÞj;jg2ðxÞjÞÞ. Therefore if we havef1ðxÞ ¼OðgðxÞÞandf2ðxÞ ¼OðgðxÞÞ, then we havef1ðxÞ þf2ðxÞ ¼OðgðxÞÞ.•If we havef1ðxÞ ¼OðgðxÞÞandf2ðxÞ ¼Oðf1ðxÞÞ, then we havef2ðxÞ ¼OðgðxÞÞ.•If we havef1ðxÞ ¼Oðg1ðxÞÞandf2ðxÞ ¼Oðg2ðxÞÞ, then we havef1ðxÞf2ðxÞ ¼Oðg1ðxÞg2ðxÞÞ.Note that iffðxÞis a sum of several terms and there is one with the largest growth rate,then that term can be kept and all others omitted, and also iffðxÞis a product of severalfactors, then constants (terms in the product that do not depend onx) can be omitted.Example 12.4Show thatfðxÞ ¼x3þ3x2þ3xþ1 isO±x3²for three pairs of witnesses.Solution(a)If we choosek¼1 (i.e.,x>1), we then have 3x3>3x2>3xandx3>1,where these inequalities are obtained by multiplying both sides ofx>1 byappropriate terms. An upper bound onfðxÞis thus as follows:0<x3þ3x2þ3xþ1<x3þ3x3þ3x3þx3¼8x3/C¼8:(b)If we choosek¼2 (i.e.,x>2), we have32x3>3x2,34x3>32x2>3x, andx38>1, where these inequalities are obtained by multiplying both sides ofx>2 by appropriate terms. An upper bound onfðxÞis thus as follows:0<x3þ3x2þ3xþ1<x3þ32x3þ34x3þ18x3¼278x3/C¼278:(c)If we choosek¼3 (i.e.,x>3), we havex3>3x2,x33>x2>3x, andx327>1,where these inequalities are obtained by multiplying both sides ofx>3 byappropriate terms. An upper bound onfðxÞis thus as follows:0<x3þ3x2þ3xþ1<x3þx3þ13x3þ127x3¼6427x3/C¼6427:Fig. 12.2showsfðxÞ ¼O±x3²for all these three pairs of witnesses.218Discrete Mathematics
Example 12.5Suppose we havefðxÞ ¼xn,wherenis a positive integer.(a)Show thatxnisO±xnþ1².(b)Show thatxnþ1is notOðxnÞ.Solution(a)Note that whenx>1, as we multiply both sides ofx>1 byxnwe havexn<xnþ1. Consequently, we can takek¼1 andC¼1 as a pair of witnesses.(b)Weuseaproof byacontradictiontoshowthatnopairofwitnessesCandkexistssuch thatxnþ1´Cxnwheneverx>k. Suppose that there is a pair of witnessesandkfor whichxnþ1´Cxnwheneverx>k. This, in turn, means that whenx>0, we havex´C(by dividing the inequality byxn). However, no matterwhatCandkare, the inequalityx´Ccannot hold for allxwithx>k. Forinstance, whenxis greater than bothkandC,it is not true thatx´Ceventhoughx>k. This contradiction shows thatxnþ1is notOðxnÞ.Whenxis a very large positive integer, the order functions satisfy the followingrelationships:Oð1Þ<Oðlog2xÞ<OðxÞ<Oðxlog2xÞ<O±x2²<Oð2xÞ<Oðx!Þ<OðxxÞ:Fig. 12.3shows a display of the growth of functions commonly used in big-Oesti-mates. Note that when we say the order of magnitude of an algorithm is a constant,Fig. 12.2Graphs for Example 12.4.Algorithms219
we mean that the execution time is bounded by a constant (i.e., it is independent of theinput sizex). If the order is linear, the execution time grows linearly (i.e., it is directlyproportional to the input sizex).Example 12.6Give a big-Oestimate forfðnÞ ¼anlogaðn!Þ þ±n2þa²loga±n2þa², wherea>1 is a real number andn>1 is an integer.SolutionWe haveanlogaðn!Þ ¼anlogað1±2±:::±nÞ ´anlogaðn±n±:::±nÞ¼anlogaðnnÞ ¼an2logaðnÞ:If we choosen>aþ1 (i.e.,k¼aþ1), we then have±n2þa²loga±n2þa²´±n2þa²loga±n2þan2²¼±n2þa²loga±ð1þaÞn2²¼±n2þa²±logað1þaÞ þloga±n2²²¼±n2þa²ðlogað1þaÞ þ2 logaðnÞÞ´±n2þa²ðlogaðnÞ þ2 logaðnÞÞ ´±n2þa²ð3 logaðnÞÞ ´±2n2²ð3 logaðnÞÞ¼6n2logaðnÞ:1101000100001004385610729Fig. 12.3A display of the growth of functions.220Discrete Mathematics
We therefore havefðnÞ ¼anlogaðn!Þ þ±n2þa²loga±n2þa²<an2logaðnÞ þ6n2logaðnÞ¼±an2þ6n2²logaðnÞ ¼ ðaþ6Þn2logaðnÞ ¼O±n2logaðnÞ²/C¼aþ6:LetfðxÞandgðxÞbe real-valued functions defined on the same set of nonnegative realnumbers. ThenfðxÞisUðgðxÞÞ, read asfðxÞisbig-OmegaofgðxÞif there are realconstantsCandksuch thatjfðxÞj ²CjgðxÞjwheneverx>k. This definition indicatesgðxÞprovides a lower bound for the size offðxÞfor large values ofx. Note thatfðxÞisUðgðxÞÞif and only ifgðxÞisOðfðxÞÞ. In short, if for sufficiently large value ofx, thevalues offðxÞare greater than those of a multiple ofgðxÞ, thenfðxÞisUðgðxÞÞ.Example 12.7Give aUðgðxÞÞestimate forfðxÞ ¼100x3³10x2þ1:SolutionForx²1, we havex3²x2/³x3´ ³x2/³10x3´ ³10x2:Consequently, we have100x3³10x2þ1²100x3³10x3¼90x3:The functionfðxÞ ¼100x3³10x2þ1 isU±x3²withk¼1 andC¼90.Example 12.8Give a big-Uestimate forfðxÞ ¼15ffiffixpð2xþ9Þxþ1, ifx²0.SolutionForx>1, we have15ffiffixpð2xþ9Þxþ1>15ffiffixpð2xþ2Þxþ1¼30ffiffixpðxþ1Þxþ1>30ffiffixp:The functionfðxÞ ¼15ffiffixpð2xþ9Þxþ1isUðffiffixpÞwithx>1 andC¼30.LetfðxÞandgðxÞbe real-valued functions defined on the same set of nonnegative realnumbers. ThenfðxÞisQðgðxÞÞ, read asfðxÞisbig-ThetaofgðxÞ, iffðxÞisOðgðxÞÞandfðxÞisUðgðxÞÞ. This definition indicatesgðxÞprovides a lower bound as well as anupper bound for the size offðxÞfor large values ofx. Note thatfðxÞisQðgðxÞÞifAlgorithms221
and only if there are real numbersC1andC2and a positive real number such thatC1jgðxÞj ´ jfðxÞj ´C2jgðxÞj, whereverx>k. In short, if for sufficiently large valueofx, the values offðxÞare bounded both above and below by those of a multiple ofgðxÞ, thenfðxÞisQðgðxÞÞ.Example 12.9Give a big-Qestimate forf xð Þ ¼5x3þ10x2log2x, ifx>0.SolutionAssumingx>1, we have the following:x>log2x>0/10x3>10x2log2x>0/5x3þ10x3>5x3þ10x2log2x>5x3/15x3>f xð Þ>5x3:Consequently, 5x3þ10x2log2xisQ±x3², whereC1¼5 andC2¼15;fork¼1:Example 12.10Give a big-Qestimate forfðnÞ ¼1þ2þ3þ:::þn,wherenis a positiveinteger.SolutionWe havefðnÞ ¼1þ2þ3þ:::þn¼nðnþ1Þ2¼0:5n2þ0:5n:Assumingn>1, we have the following:0:5n2>0:5n>0/0:5n2þ0:5n2>0:5n2þ0:5n>0:5n2/n2>f nð Þ>0:5n2:Consequently, 0:5n2þ0:5nisQ±n2², whereC1¼0:5 andC2¼1, fork¼1:IffðxÞ ¼anxnþan³1xn³1þ:::þa0is a polynomial of degreen²0, wherenis aninteger, all coefficients are real numbers andans0;then for large values ofx,wehave the following:8>><>>:f xð Þ ¼OðxsÞfor all integerss²nf xð Þ ¼UðxrÞfor all integersr´nf xð Þ ¼QðxnÞ222Discrete Mathematics
Note that big-Oh, big-Theta, and big-Omega notations can be extended to functionsin more than one variable. For example, for functions in two variables and some wit-nesses, we have the following:8>><>>:fðx;yÞ ¼Oðgðx;yÞÞ/jfðx;yÞj ´C1jgðx;yÞjforx>k1andy>k2fðx;yÞ ¼Uðgðx;yÞÞ/jfðx;yÞj ²C2jgðx;yÞjforx>k1andy>k2fðx;yÞ ¼Qðgðx;yÞÞ/C2jgðx;yÞj ´ jfðx;yÞj ´C1jgðx;yÞjforx>k1andy>k212.5 Sorting AlgorithmsAn entry in a database is a 2-tuple whosefirst component is a key and the second compo-nent is the corresponding data. In other words, a key in a database is a value from an or-dered set, which is used to store and retrieve data. The process of arranging a collection ofdatabase entries into a sequence that conforms to the order of their keys is calledsorting.Asorting algorithmis an algorithm that puts elements of a list in a certain order, such as anascending order or a descending order.A sorting algorithm puts the elements of a list in a certain place or rank according tokind, class, or nature. Such an arrangement in the context of discrete mathematics andcomputer science is generally in alphabetical or numerical order. Sorting examplesmay include words in a dictionary, names in a telephone directory, email addresses ina contact list, cities according to the sizes of their populations, and countries based ongross domestic product. A major advantage of using a sorted sequence of elements ratherthan an unsorted sequence of elements is that it is very much easier tofind a particularelement, especially when the sequence is quite long.A significant portion of computing resources is often devoted to sorting problems.There are numerous sorting algorithms. Some algorithms are easier to implement, andsome are more efficient either in general or for a set of particular inputs. Each algorithmhas certain benefits and drawbacks. The focus of this section is on comparison-sortingalgorithms, meaning sorting algorithms that can sort items of any type for which aless-than or equal-to relation is defined (i.e., which of two elements should occurfirstin thefinal sorted list). A comparison sort cannot perform better thanOðnlog2ðnÞÞonaverage.Table 12.1presents the complexities of various sorting methods.Thebubble sortis a simple but not very efficient sorting algorithm. In the bubble sort,smaller elements bubble to the top and larger elements sink to the bottom. It successivelycompares adjacent elements in the list and swaps them if they are not in the right order.Thefirst element is compared with the second element, the elements are interchanged ifthe second element is smaller than thefirst one, otherwise no swapping is done. Then thecurrent second element is compared with the third element, those elements areAlgorithms223
interchanged if the third element is smaller than the second one. This process of inter-changing a larger element with a smaller one following it starts with thefirst elementand continues to the last element for a full pass. This procedure is repeated throughseveral passes until the sort is complete. In a list ofnelements, at the end of theithpass, theilargest elements are correctly placed at the end of the list, where 1´i´n.The bubble sort takesn³1 passes to sort a list ofnelements. There arenelements,and for each element,n³1 comparisons are made, this thus leads to a total timecomplexity ofO±n2².Theinsertion sortis a simple sorting algorithm that builds thefinal sorted list oneelement at a time. It is less efficient than most sorting algorithms unless the number ofelements in the list is modest, say about 50 or less. The insertion sort begins with the sec-ond element and compares it with thefirst element and inserts it before thefirst elementif it does not exceed thefirst element and after thefirst element otherwise. Then the thirdelement is compared with thefirst element, and if it is larger than thefirst element, it iscompared with the second element, and it is inserted in the correct position among thefirst three elements. The insertion sort iterates through the list and removes one elementper iteration,finds the place that the element belongs to, and then places it there. Theresulting list afterkiterations has the property where thefirstkþ1 elements are sorted.The insertion sort requires at mostn³1 comparisons andn³1 swaps. At each step, thenumber of comparisons and swaps decreases by one. Using the arithmetic progression,this gives rise to a total ofnðn³1Þcomparisons. This thus leads to a total time complexityofO±n2².Themerge sort, a recursive sorting algorithm, focuses on how to merge together twopresorted lists such that the resulting list is also sorted. A merge sort proceeds by iterativelysplitting lists into two sublists of equal or almost equal numbers of elements until each sub-list contains one element; a list of one element is considered sorted. The merge sort thensuccessively merges pairs of sublists, where both sublists are in increasing order, to producea new larger sublist with elements in increasing order. This continues until the original listis put into increasing order. The merge sort is a recursive algorithm and time complexityTABLE 12.1Complexities of various sorting methods.Sorting methodsWorst-case time complexityAverage-case time complexityBubble sortInsertion sortMerge sortQuicksortSelection sortO±n2²O±n2²OðnlognÞO±n2²O±n2²O±n2²O±n2²OðnlognÞOðnlognÞO±n2²224Discrete Mathematics
can be expressed as the following recurrence relationfðnÞ¼2f±n2²þOðnÞ, whereOðnÞis what it takes to combine the sublists. The solution, that is, the time complexityof the merge sort, is thenOðnlognÞ.Thequicksortis an efficient sorting algorithm employing a divide-and-conquer strat-egy. It begins by selecting a pivot element, usually thefirst element from the list, and par-titioning the other elements into two sublists. One sublist consists of all elements in thelist less than the pivot and the other consists of all elements greater than the pivot. Thenthe pivot is put at the end of thefirst list as itsfinal resting place. The two sublists are thenrecursively sorted until all sublists contain only one element. The sorted list can then beobtained by combining the sublists of one item in the order they occur. The quicksortrequiresOðnÞcomparisons to scan through allnelements to divide them into two sub-lists. The quicksort is a recursive algorithm. In a balanced division, we have the average-case time complexity that can be expressed as the following recurrence relationfðnÞ¼2f³n³12´þOðnÞ, whose solution is thenOðnlognÞ. In the worst-casescenario (i.e., the pivot is the smallest or the largest in the list), the corresponding recur-rence relation isfðnÞ ¼2fðn³1Þ þOðnÞ, whose solution is thenO±n2².Theselection sortis a very simple but inefficient sorting algorithm. The algorithmproceeds byfinding the smallest element in the unsorted list and moves it to the topof the sorted list. Then the least element among the remaining elements of the unsortedlist is found and moved to the second position of the ordered list. This procedure isrepeated until the entire list has been sorted. The selection sort requires atn³1 compar-isons to select the smallest element and swap it to make it into thefirst position, andn³2comparisons to select the second smallest element and swap it, and so on. Using the arith-metic progression, this gives rise to a total ofnðn³1Þ2comparisons. This thus leads to a totaltime complexity ofO±n2².Example 12.11Sort the listf7;6;1;9;5gusing the following sorting algorithms:(a)Bubble sort.(b)Insertion sort.(c)Merge sort.(d)Quicksort.(e)Selection sort.Solution(a)In each step of a pass, elements in boldface are being compared.Algorithms225
1st pass/8>>>>>>>>>><>>>>>>>>>>:7;6;1;9;56;7;1;9;56;1;7;9;56;1;7;9;56;1;7;5;9;2nd pass/8>>>>>>>>>><>>>>>>>>>>:6;1;7;5;91;6;7;5;91;6;7;5;91;6;5;7;91;6;5;7;9;3rd pass/8>>>>>>>>>><>>>>>>>>>>:1;6;5;7;91;6;5;7;91;5;6;7;91;5;6;7;91;5;6;7;9;4th pass/8>>>>>>>>>><>>>>>>>>>>:1;5;6;7;91;5;6;7;91;5;6;7;91;5;6;7;91;5;6;7;9Note that the bubble sort needs one whole pass (i.e., the fourth pass) withoutany swap to know it is sorted.(b)The insertion sort begins with 6 and compares it with 7. It inserts it before 7,as it does not exceed 7. Then, 1 is compared with 6, and as it is smaller than 6,it is inserted before 6. Next, 9 is compared with 1, as it is larger than 1, it iscompared with 6, as it is larger than 6, it is compared with 7, as it is larger than7, it is inserted after 7. Finally, 5 is compared with 1, as it is larger than 1, it iscompared with 6, as it is smaller than 6, it is inserted after 1 and before 6. Notethat the following shows the required steps, where the element under consid-eration in each step is in boldface:7;6;1;9;5/6;7;1;9;5/1;6;7;9;5/1;6;7;9;5/1;5;6;7;9.(c)Fig. 12.4shows the merge sort, where the top part is the split and the bottompart is the merge.(d)Employing the quicksort, thefirst element (i.e., 7) is selected, and we partitionthe other elements into two sublists. Thefirst sublist consists of all elements inthe list less than 7 (i.e., 6, 1, 5), and the other consists of all elements greaterthan 7 (i.e., 9). Then 7 is put at the end of thefirst sublist as itsfinal restingplace. The two sublists are then recursively sorted the same way, that is, 6is selected in thefirst sublist and the other elements in that sublist arecompared with 6 (i.e., 1, 5), and 9 is selected in the second sublist. Notethat the following shows the required steps, where the element under consid-eration in each step is in boldface:7;6;1;9;5/6;1;5;7;9/1;5;6;7;9/1;5;6;7;9226Discrete Mathematics
(e)The selection sortfinds 1, as it is the smallest element in the unsorted list, andmoves it to the top of the sorted list. Then the least element among theremaining elements of the unsorted list (i.e., 5) is found and moved to the sec-ond position of the ordered list. This procedure is repeated until the entire listhas been sorted. Note that the following shows the required steps, where theelement under consideration in each step is in boldface:7;6;1;9;5/1/1;5/1;5;6/1;5;6;7/1;5;6;7;9/1;5;6;7;9:12.6 Search AlgorithmsSearchingis the process of locating an element in a list. Asearch algorithmis an algorithmthat involves a search problem. Searching a database employs a systematic procedure tofind an entry with a key designated as the objective of the search. A search algorithm lo-cates an elementxin a list of distinct elements or determines that it is not in the list. Thesolution to the search is either the location of the elementxin the list or 0 ifxis not onthe list. We now briefly introduce two well-known search algorithms whose worst-caseand average time complexities are presented inTable 12.2.TABLE 12.2Complexities of various search methods.Search methodsWorst-case time complexityAverage-case time complexityLinear searchBinary searchOðnÞOðlognÞOðnÞOðlognÞFig. 12.4Merge sort for Example 12.11.Algorithms227
Thelinear search, also known as thesequential search, is the simplest search algorithm.It is an algorithm, based on the brute-force algorithmic paradigm, that scans the elementsof a list in sequence in search ofx, the element that needs to be located. A comparison ismade betweenxand thefirst element in the list, if they are the same, then the solution is1. Otherwise, a comparison is made betweenxand the second element in the list, if theyare the same, then the solution is 2. This process continues until a match is found and thesolution is the location of the element sought. If no match is found, then the solution is 0.Linear search is applied on unsorted or unordered lists consisting of a small number ofelements. Becausencomparisons are required tofindx,the linear search has a timecomplexity ofOðnÞ, which means the time is linearly dependent on the number of el-ements in the list.The list of data in a binary search must be in a sorted order for it to work, such asascending order. This search algorithm, which is quite effective in large sorted array,is based on the divide-and-conquer algorithmic paradigm. Abinary searchworks bycomparing the element to be searched with the element in the middle of the array ofelements. If we get a match, the position of the middle element is returned. If the targetelement is less than the middle element, the search continues in the upper half of thearray (i.e., the target element is compared to the element in the middle of the upper sub-array), and the process repeats itself. If the target element is greater than the middleelement, the search continues in the lower half of the array (i.e., the target element iscompared to the element in the middle of the lower subarray), and the process repeatsitself. By doing this, the algorithm eliminates the half in which the target element cannotlie in each iteration. Assuming the number of elements isn¼2k(i.e.,k¼log2n), atmost 2kþ2¼2 log2nþ2 comparisons are required to perform a binary search.Binary search is thus more efficient than linear search, as it has a time complexity ofOðlognÞ:The worst case occurs whenxis not in the list.Example 12.12Consider an array consisting of these integers: 2, 3, 5, 7, 23, 19, 17, 13, 11, 29,31, 37.(a)Use the linear search tofind 17.(b)Use the binary search tofind 17.Solution(a)Thefirst element of the array, that is, 2 is compared with 17, as they are notthe same, 3 is compared with 17, as they are not the same, 5 is compared with17; this process continues until 17, which is in the list, is found. Then the so-lution is 7, as 7 is the location of 17 in the list.(b)In order to apply the binary search, wefirst need to sort the list. The sortedlist, in ascending order, is then as follows: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31,37. As the number of integers in the list is 12, there is no middle integer.228Discrete Mathematics
Therefore the sorted list is split into two sublists, each consisting of six inte-gers, namely, 2, 3, 5, 7, 11, 13 and 17, 19, 23, 29, 31, 37. Then compare17 and the largest integer in thefirst list. Because 17>13, the search for17 must be restricted to the second sublist, namely, 17, 19, 23, 29, 31, 37.Next we split this sublist into two smaller sublists, each with three integers,namely, 17, 19, 23 and 29, 31, 37. Then, compare 17 and the largest integerin thefirst sublist. Because 23>17, the search for 17 can be restricted to thefirst sublist. Next we compare 17 to the middle integer in this sublist, namely,19, as 19>17, we split this sublist into two integers, one smaller than 19 (i.e.,17) and the other larger than 19 (i.e., 23). The search has been narroweddown to one term, we thus compare 17 and 17, and 17 is thus located inthe sorted list.Exercises(12.1)Determine the increase in complexity for each of the following functions whenthe input sizenis increased tonþ1, assumingnis a very large positive integer.(a)fðnÞ ¼logn.(b)fðnÞ ¼n.(c)fðnÞ ¼nlogn.(d)fðnÞ ¼nm, integerm>1:(e)fðnÞ ¼2n.(f)fðnÞ ¼n!.(12.2)Apply the binary search algorithm tofind 72 in the following sorted sequence:1;8;9;13;22;27;36;47;49;72;81;100;121;144;150(12.3)Determine the big-Oh and big-Theta of the following function:fðxÞ ¼x7þx4þx3þx2þxþ1x3þ1:(12.4)Show that 8x4isO±x5².(12.5)Determine the big-Oh of the following function, wherenandmare both positiveintegers.±nlognþn2²m:Algorithms229
(12.6)Describe how the number of comparisons used in the worst-case changes in eachof the following search algorithms when the size of the list increases fromntomn,wheren>0 andm>0 are positive integers.(a)Linear search.(b)Binary search.(12.7)Show thatfðx;yÞ ¼±x2þy2þxyþylogx²4isO±x8y8².(12.8)Describe how the number of comparisons used in the worst-case changes in eachof the following sorting algorithms when the size of the list increases fromntomn,wheren>0 andm>0 are positive integers.(a)Bubble sort.(b)Insertion sort.(12.9)Arrange the following functions in a list so that each function is big-Oof the nextfunction.2n;n100;ðlognÞ4;ffiffin3plogn;10n;ðn!Þ2:(12.10)Noting thatfðnÞisOðgðnÞÞ, determinegðnÞ, wherefðnÞis as follows:fðnÞ ¼1kþ2kþ:::þnk;wherekandnare both positive integers.230Discrete Mathematics