If a problem does take time that is exponential in its input size, then that means it can in practice only be solved for small instances. An algorithm that runs in some large polynomial time, like N to the thousandth power, is no more practical than one that runs in time two to the N. We introduce the idea of a Turing Machine that is time-bounded. It can only run for time that is a known function of its input size before it has to stop.
If a problem does take time that is exponential in its input size, then that means it can in practice only be solved for small instances. An algorithm that runs in some large polynomial time, like N to the thousandth power, is no more practical than one that runs in time two to the N. We introduce the idea of a Turing Machine that is time-bounded. It can only run for time that is a known function of its input size before it has to stop.
If a problem does take time that is exponential in its input size, then that means it can in practice only be solved for small instances. An algorithm that runs in some large polynomial time, like N to the thousandth power, is no more practical than one that runs in time two to the N. We introduce the idea of a Turing Machine that is time-bounded. It can only run for time that is a known function of its input size before it has to stop.
problems. These problems are decidable but co-locally they are said to be problems that take at least exponential time as a function of their input size. The reality is a bit different. But there are problems which there's an overwhelming amount of empirical evidence that these problems take exponential time, although no solid proof of that belief. If a problem does take time that is exponential in its input size, then that means it can in practice only be solved for small instances. Suppose to be concrete that the time it takes to solve an instance of size N is two to the N. And doubling the speed of machines makes essentially no difference in how large an instance you can solve. It adds one to the size you can solve in a fixed amount of time. Using a thousand machines instead of one has the effect of adding ten to the size N. And using a million machines, each one thousand times faster than today's machines adds thirty to N. You never get to really big sizes of problem instances that you can solve. As a result, it is generally accepted that in order to a solution to Problem to be considered usable in practice. It has to run in less than exponential time, and in particular in polynomial time for some large polynomial. Well an algorithm that runs in some large polynomial time, like N to the thousandth power, is no more practical than one that runs in time two to the N. You find in practice that if a problem has a polynomial algorithm at all, then it has an algorithm that runs in some low degree polynomial, like N squared or N cubed at the most. In this lecture we introduce several important preliminary concepts. We introduce the idea of a Turing machine that is time-bounded. It can only run for time that is a known function of its input size before it has to stop and tell us whether it accepts or rejects. We introduce the Class P of problems or languages, it's the same thing of course, that can be solved by a Turing Machine that runs in polynomial time as a function of its input size. We also meet the Class NP which is problems that can be solved by a Turing Machine that is non-deterministic but has a [inaudible] time bound along each branch. Finally, we'll learn about polynomial time reductions, which are reductions where the transducer runs in time which is polynomial in its input size. These are used to show one problem intractable by reducing a known intractable problem to it. [sound] We say a Turning Machine is T of N, time-bounded where T of N is some function of N like N squared or two to the N. If given an input of length N the machine always halts at most T of N steps. Okay? We allow the Turing Machine to have several tapes. In some circumstances we allow the Turning Machine to be non-deterministic, although in that case we will specify that it is a [inaudible] time-bounded non-deterministic Turning Machine. Also in that case will mean that any, any sequence of moves of the non-deterministic machine is no longer than T of N. In practice, a deterministic multi-tape Turing Machine is close to the idea of an algorithm that runs in time proportional to T of N or big O of T of N. That is why some algorithms take longer in a Turing Machine, even multi-tape, than on a real computer these are rare. Moreover when there is a difference, the difference tends to be small A turn machine and the set to be polynomial time bounded, if it is time bounded by any polynomial. It could be linear, quadratic, cubic or into the thousandth power, as long as it is some polynomial. The languages that are accepted by polynomial time bounded turn machines form the class P. Now P is defined formally in terms of Turing Machines, but it could just as well have been stated as polynomial time on a real computer. The reason, which we address on the next slide, is that if an algorithm runs in some polynomial time on a computer, then it will run polynomial time on a multi-tape Turing Machine, or even a one tape Turing Machine, although the degree of the polynomial may be higher in some cases. That is, we saw a way to simulate a name value store by a computer. That is the part of a real computer that takes the most time when simulated by a Turing Machine. But if a computer runs for on the order of T of N steps then it can store or retrieve more than T of N items in its memory. A Turing Machine can simulate one look up or insert into a name value store and a number of steps that is proportional to the length of the tape that holds the store. But that length is proportional at most to the number of steps the computer has taken, which is T of N. And thus the Turing Machine takes at most T squared of N of its own steps. If T of N is a polynomial, then so is T squared of N. The exponent grows of course, a cubic algorithm on a computer might take time proportional to N to the sixth on a Turing Machine, but it's no worse than that. And since we're trying to divide the world of problems into those that have polynomial algorithms and those that don't, we can think Turing Machine or computer whichever is more convenient. As you might expect, when simulating a program it's best to simulate a Turing Machine, but when devising an algorithm it's best to think about a computer program. Here are two examples of problems or languages, which is the same thing, in the Class P. For each context-free grammar G there is an algorithm, the CYK algorithm, that takes an input string W and tells whether W is in the language. There's many times that this lag-, algorithm is o of N cubed. The second problem I want to talk about is finding a path in a graph. Here we're given a directed graph, that is a list of its nodes and arcs. We are also given one node that is the source node X and another is the sync node Y. The answer's yes, that there's a path in the graph from the source to the sync. Graphs must be coded in a [inaudible] alphabet, which should not be hard to see, represent the [inaudible] node by N followed by I in binary in represent an arc by a pair of nodes that tail and the head of the arc. Used two special symbols to indicate the source and sync nodes. Note that there are M nodes, it takes order log M space to represent one node, so N, the input length, is actually somewhat greater than the number of nodes and arcs, but the difference is unimportant since we are only worrying about polynomial versus not polynomial. Depth for a [inaudible] answers this question in time that is linear in the number of nodes and arcs. On a turing machine you might need order, order N squared steps, since for one step of the depth for [inaudible], you have to locate on the input the arcs with a given node as the tail. That could require that you run all along the tape just to simulate one computer step. But N squared is still a polynomial so as far as membership in P is concerned N squared is just fine. And just to make sure, when we talk about polynomial time, we talk about every running time that is less than some polynomial. That is the definition of B only requires that the language be accepted by some [inaudible] machine, whose running time is bounded above by some polynomial. For example there are many algorithms that run in time, like order N log N, but that is less than N squared so the problems solved by algorithms like this are surely empty. Before proceeding, I want to examine in detail a problem that seems to be in P but really isn't. And I want you to understand why, this is really important in understanding what the class P really means. The problem called knapsack is this. We're given a list of N positive integers. The answer to this instance of nap stack is yes, if and only if we can partition the integers into two groups whose sums are equal. For example, if the integers are one, two, three and four Then i can partition them into one in four in one group, and two and three in the other group, and the sums in each group will be equal. Incidentally the problem is called Nap Sack because of the view that the integers await the items and two hikers want to divide the items between their two nap sacks, so each carries equal weight. At first glance we can solve an abstract problem by a polynomial time dynamic programming algorithm. That is, we maintain a table of all the differences and sums we can achieve by partitioning the first J minus one integers. When we incorporate the Jth integer, we take each possible difference and both add and subtract the Jth integer thus getting two new possible differences. After looking at all integers we see if zero is a possible difference. To be more precise, for the basis we consider none of the integers. Then the table That's true for zero difference and false for all the other differences. For the induction, suppose we have a table for the first J minus one integers. We build a new table to reflect the partitions of the first J integers, initially each entry in the new table is false. But suppose the Jth integer is [inaudible] of J. For each difference, M that was true in the old table set the entries for M plus I's of J and also M minus I of J to be true in the new table. Lets compute the running time of this algorithm as a function of the sum of the integers. Lets say that some is S. We need order S space to construct a table for one value of J, since the differences must be in the range minus S to plus S. And it only takes order S time to construct each table from the previous one using a real computer, maybe it's order S squared on a Turing machine because you have to move the head a long distance to write each entry. But again, when designing algorithms and worrying only about whether something is polynomial time or not, real computer is the right model to think about because the programming details are generally easier. Okay note that N equal to or less than S, since the integers are each positive. That is the sum of the integers is at least equal to the number of integers on the list. Thus we can build a table that corresponds to the set of all the integers in order S squared time. S for each of N different tables. We then look at this table and see if zero is true. If so, the answer to the knapsack instance is yes. And otherwise it is no. However that conclusion is actually deceptive. Although it is true that we just described an algorithm that runs in time no more than the square of the sum of the integers and that algorithm really does solve the nap sack problem. It doesn't tell us that the nap sack problem is in P and in fact it is surely not in P as we shall see later. The reason this algorithm doesn't show knapsack to be in P is that membership in P requires that the algorithm runs in time polynomial in the input size. But we can't just define input size to be the sum of the integers in the input. The input size is always the number of cells it takes to write the input on a Turing Machine tape. For the knapsack problem, this input length is not necessarily polynomial in the sum of the integers, as we shall see on the next slide. The longest input length occurs if we have any integers, each of whose values is about two to the N. If we write the integers in binary, the input to the Turing Machine is order N squared in length. But a table then requires about two to the N entries and at least that much space to write down. That is the sum of all N integers can be around N times two to the N. Mccath can construct one table in time less than its length so the total time of the algorithm is on the order of N squared times two to the N But the input size is N squared and N squared two to the N is not a polynomial function of input length. By the way, we usually like to have N be the actual input size for the Turing Machine. So if we substitute N for N squared we can say the inputs of size N leads to an algorithm that takes time proportional to N times two to the squared root of N. That is still not a polynomial. Thus the dynamic programming algorithm we described, while it is really a good algorithm when the integers of limited size, does not show the knapsack problem to be in the class P. There is another problem which we can call pseudo-knapsack. The question is the same, but the integers are represented in unary not binary. That is interger I is represented by I ones followed by some Marcuss symbol toseparateeintegerss. This problem is in P, and the dynamic programing algorithm proves that. But it is not the classical nap sack problem where integers are represented in binary, the sort of rational way to represent large integers. The second important class of languages for our story is NP, a nondeterministic polynomial class. Np is defined in terms of nondeterministic Turing Machines. The running time of a nondeterministic Turing Machine is the maximum number of moves it takes along any branch, that is making any sequence of choices. If there's a polynomial bound on that time then the non-deterministic machine is said to be polynomial time-bounded. And the language or problem it accepts is said to be in the class NP. For example, the standard version of knapsack where integers are represented in binary is an NP. The non-deterministic polynomial time algorithm that solves this is fairly simple. First it uses its non-determinism to guess a partition of the integers into two subsets. This can be done in time that is linear in the input length using two extra tapes for the two subsets. Then sum the subsets and compare. Say yes if this partition yields two equal sums. And this part can truly be done in time that is quadratic in the input size and can be done in linear time if you're clever and use a few extra tapes. Thus standard knapsack is an NP. Note that this fact doesn't suggest a deterministic polynomial time algorithm, since it may take exponential time to simulate the non-determinism of the Turing Machine. Are P and NP really the same class of languages? That is, can any problem that is solved by a non-deterministic Turing machine in polynomial time also be solved by some deterministic Turing Machine in polynomial time, even if the degree of the polynomial is higher? This question was posed by Steve Cook in 1970. At first it didn't seem all that hard or unlikely. After all nondeterministic [inaudible], [inaudible] can be simulated by deterministic [inaudible], [inaudible], even though the number states might grow and deterministic Turing Machines can simulate nondeterministic ones. But the problem is proving to be very, very difficult, and mathematicians who once neared the question and assumed it was easy because computer scientists have thought it up, now recognize it as one of the most important mathematical question, perhaps the most important unsolved question. However there are thousands of problems that are in N P but for which no algorithm N P has been found. And unfortunately neither is there proof that these problems are not N P. What we do have is a linkage of the large class of problems called N P Complete Problems which we discuss on the next slide. What we do know is then that either all these problem are in P or none of them are. So the mutually enforce the notion that none of them are since many have been worked for decades and no polynomial time algorithm for any of them has been found. So we're going to address the question of whether P equals N P by identifying complete problems for N P. We say a problem is NP complete if the following is true about the problem. If the problem is NP then P equals NP, that is every problem in NP is also NP. It turns out that almost every problem that is known to be an N P, but is not known to be N P is N P complete. There is only one well known exception, [inaudible]. Given two graphs is there a one to one matching of nodes between the two graphs that makes the graphs identical? This problem is known the be N P just guess matching the two nodes and check that the right edges exist. But there is no polynomial time algorithm known and neither is there a proof that the graph is M P complete But [inaudible] is an exception to what appears to be an almost general rule, if it is an NP and it is not known to be NP, then it is NP complete. While the definition of NP complete is merely states that there has to be some way proving that P. Equals NP from the assumption that the problem is NP. There is a standard way of making such proofs and it appears to be sufficient for all the NP complete problems we know about. This method involves reductions of the type we talked about in Turing Machines in general, but with the condition that the transducer in polynomial time is a function of its input size. Intuitively a complete problem for a class embodies, in some sense, every problem in the class. For example post correspondence problem embodies every Turing Machine even though it is hard to see PCP as. Computation. It only seems to be about concatenating strings in constrained ways. So it might surprise you to know that each NP complete problems, such as nap sack, embodies all nondeterministic polynomial time computation, even though the nap sack problem seems to be about anything but computation. So in order to show problem L to be NP complete we have to show that every problem in NP is somehow embedded in L. We need a transformation from every problem in NP to L and this transformation has to be sufficiently fast that if we had to deterministic polynomial timed algorithm for L, then we could use it to build a deterministic polynomial timed algorithm for each problem in NP. We are going to define a polynomial time transducer. Notice that people frequently shorten polynomial time to poly time, and we will start doing that to, So a poly time transducer is a deterministic Turing Machine in that it takes an input of length N, runs for some polynomial number of steps, P of N. And produces an output on an output tape. It is important to observe that although we do not restrict the output length since the Turing Machine only runs P of N steps. It could not write more than P of N symbols, thus the length of the output of the polynomial time transducer is always polynomial in the length of its input. Here is a picture of a poly time transducer. He can have any fixed number of tapes, one is the input tape, and one is the output tape. We argued on the last slide that the output length was the polynomial in the link of the input, but the real constraint on the poly time transducer is on how long it runs. It is not acceptable to have it run for time that is exponential in its input length, even if the output is short. Consider two languages or problems say LMN. We say L is poly-time reducible to M if there is a poly-timed translucor T that takes and inputs W that is in the instance of L, produces an output of X that is in the instance of M and the adds are on W is the same as the answer to M on X. That is W is L only if X is in M. Here is a picture to help us remember what a poly time reduction does. On the left is set of strings for the alphabet of L divided into those that are in L and those that are not in L. On the right is the set of strings over the alphabet of M, also divided by its compliment. In the middle is the poly-timed translucor of T. Every string in L is transformed by T into a string that is in M. There can be strings in M that are not the target of strings in L. And every string not in L but over the alphabet of L is transformed by T into a string that is over the alphabet of M, but is not in M. Again there can be compliments of strings that are not the target of any strings in the compliment of L. Formally we say a problem or language of MP complete if for every language L or MP there is a poly timed reduction from L to M. An important consequence of the fact that M is MP complete is that if M has a poly timed algorithm then so do every L in MP, that is the classic language of P or MP are the same, or P equals MP. Notice that earlier we suggested that the definition of MP completeness was simply that the language had this property. Steve Cooks original definition of MP complete was exactly that. And it is often referred to as Cooks completeness. Cook concentrated on showing one particular problem- the question of whether an expression of proposition logic was satisfiable that is made true by some assignment of truth values to the propositional variables. But shortly after Cook wrote his original paper on MP completeness, Dick Carp wrote another paper that showed many of the classical problems that have been puzzling mathematicians for centuries where MP complete. Carp used only poly-timed reductions to the problems Cook had completed. Since then it is generally accepted that the preferred definition of NP complete and this is the one we gave here, the existence of poly-timed ductions. To make the distinction this notion of MP completeness is often called Carp Completeness. So here is the plan for proving certain problems to be NP complete. Here is all of NP, sad the satisfiable problem for propositional logic that we just discussed is one problem in NP. Cook's theorem is that every problem and then MP reduces in poly-timed to the sad problem. So sad is the first known MP complete problem. Cook also proved a restrictive form of sad called Three Sad to be MP complete by reducing sad to three sad. We'll learn about the three sad restriction is shortly, but in brief it is sad restrictive to expressions that are the and of clauses with three literals per term. A literal is a variable or a negated variable and a clause is the or of literals. Then from three sad we do poly timed reductions either directly or Indirectly. Each problem we can reach from sack by a chain of poly-time reductions is thus proved NP complete. But before we embark on this quest, let's make sure that a polynomial time reductions work in the sense that they let us draw the desired conclusion about all of NP reducing to the target problem. So suppose M has a poly-time algorithm, say running time Q of N for some polynomial Q. Let that be a poly-time transducer T from some problem L to M. And let the time taken by T be P of N for some polynomial. [inaudible] the output of T given an input of length N is at most of length P of N, so when we run the algorithm for N on this input of length P of N, the algorithm takes time Q of P of N. Note that a polynomial of a polynomial is a polynomial. The degrees of the polynomial are multiplied but it's still a polynomial. We claim there is a poly-time algorithm for L, given an input W of length N for L, apply the transducer T to W. The result is Now put X at length of P and N and more importantly T takes time only P of N to produce this output. Apply to X the algorithm to tell whether it is in M. As we observed in the previous slide this part takes time Q of P of N. And in return the answer to W whatever the N algorithm says about X. The total time of this algorithm is P of N plus Q of P of N which is polynomial of P and QR. It is the correct algorithm because that the fact that T is a poly-timed translucor from L to M says that the answers to input W and output X are the same.