Research article CAD-guided robot motion planning

Weihua Sheng Ning Xi Mumin Song and Yifan Chen

Introduction
CAD/CAM technology has been extensively adopted in the manufacturing industry in the last few decades. Although CAD and CAM technology, individually, have been fully developed, the connection between them, or manufacturing process planning, has not achieved equal success. For example, in robot path planning, intensive human intervention is still indispensable, which usually results in low efficiency and errors. This problem is aggravated in small batch productions. In the last few decades, a lot of research has been done in computer aided process planning (CAPP) (Zhang and Alting, 1994). Research has shown that to effectively bridge CAD and CAM, it is essential for the system to be able to obtain higher level information and representations from low level representation. Based on the higher level information, there exist two different approaches for process planning: (1) variant; and (2) generative (Marefat and Kashyap, 1992). In the variant approach, a group technology is used to select a generic process plan from a database of standard process plans developed for each part family. The selected generic plan is then customized interactively for the particular part. If the plan does not exist for the family, the database is enhanced by an experienced process engineer. This approach, in its nature, is semi-automatic since it still needs human intervention. On the other hand, generative process planning systems are relatively new developments, however, they are far from maturity. Generative planning involves constructing plans from basic input data based on general rules for manufacturing (Marefat and Kashyap, 1992). It is our belief that the generative approach is promising for achieving full automation in process planning in spite of the fact that it is still not matured. This paper tries to solve, specifically, the robot motion planning problem in surfacebased manufacturing. A typical application scenario is: a robot holds a tool or a sensor to carry out some tasks on the part surfaces, such as inspection, painting, fiber-glass deposition etc. To achieve full automation, CAD data and tool or sensor model are input to the robot motion planner. Robot paths, which satisfy the specified task constraints, are generated. Figure 1 shows the block

The authors Weihua Sheng and Ning Xi are both based at the Electrical and Computer Engineering Department, Michigan State University, East Lansing, MI, 48824, USA. Mumin Song and Yifan Chen are both based in the Scientific Research Labs, Ford Motor Company, Dearborn, MI 48121, USA. Keywords Robots, Painting, CAD Abstract This paper presents a new method to automate robot motion planning in automotive manufacturing environments. A general framework is developed for CAD-guided robot motion planning. The problem is formulated as a constraint-satisfying problem of tool configurations or, robot hand poses. Two types of robot motion are considered: discrete motion, or point to point motion, and continuous motion. Triangular facets are used to approximate the part surfaces. A pre-partition process decomposes the complex part surfaces into several simple, easy-to-solve patches. For each patch, robot hand poses are determined to satisfy certain task constraints. In this paper, the approach is applied to two applications: vision sensor planning and spray painting gun path planning. It is our belief that more robot planning applications in manufacturing can benefit from this method. Electronic access The research register for this journal is available at http://www.mcbup.com/research_registers The current issue and full text archive of this journal is available at http://www.emerald-library.com/ft
Industrial Robot: An International Journal Volume 28 . Number 2 . 2001 . pp. 143151 # MCB University Press . ISSN 0143-991X

143

CAD-guided robot motion planning

Weihua Sheng, Ning Xi, Mumin Song and Yifan Chen

Industrial Robot: An International Journal Volume 28 . Number 2 . 2001 . 143151

Figure 1 CAD-guided robot motion planner

diagram of the CAD-guided robot motion planner. Basically, there are two different types of robot movement in these applications. One is discrete movement, or point-by-point movement, that is, the robot performs certain tasks on a number of discrete configurations. An example is to use an eye-in-hand setup to inspect a part. Another is continuous movement. This is usually seen in painting and polishing applications where the robot executes certain tasks during the movement along the whole path.

to obtain global knowledge about the part surfaces, or high level information about the geometry of the part. 2-D triangular tessellation, which is a scheme popularly used in CAD applications, is adopted in our system. The advantages of adopting triangular tessellation are: . It is possible to analyze the global geometric structure of the part surfaces using the triangles, essentially, the normal vector, the area and the connectedness between triangles. . Each triangle is a simple geometric entity and it is attractive for algorithm implementation. On the other hand, the discretization error introduced in tessellation can be easily controlled to be within the tolerance of the applications. Tool model A typical tool model in surface-based manufacturing is shown in Figure 2. Generally, a tool has a working range which confines the tool to be within a certain distance range from the surface to be worked on. In most of the surface-based tasks, for a point on the surface to be admissible to certain tool configuration, there exist certain constraints imposed on the position and normal of the point. To characterize the normal of a point p, the angle  between the central axis z of the tool and the normal of point p is adopted. Decomposition of compound surfaces Most of the parts in sheet metal manufacturing share a common characteristic in their surface geometry. They have several
Figure 2 A general tool model

Problem statement
For simplicity, the sensor and the tool are not explicitly distinguished. In a wider sense, a sensor is a kind of tool, so the ``tool'' used in this paper takes this wider meaning. A tool configuration is defined as a set of parameters which uniquely describe a status of the tool, which, usually, is a 6-D vector representing the position and orientation of the tool. The tool path is defined as an ordered sequence of tool configurations. The tool planning problem can be stated as:
Given a CAD model of the sheet metal part, design a set of tool configurations or a continuous tool path, such that the specific task can be carried out satisfactorily over all the surfaces of the part.

CAD model and tool model

CAD model Parametric surfaces are extensively used in CAD modeling and they usually exhibit certain continuity and smoothness most of the parametric surfaces have low curvatures. Although parametric representation is mathematically accurate, its local nature brings difficulties for tool planning due to the fact that each parametric surface has its own variables, different from those of other surfaces (Lai and Wang, 1994). For tool planning, it is desirable 144

CAD-guided robot motion planning

Weihua Sheng, Ning Xi, Mumin Song and Yifan Chen

Industrial Robot: An International Journal Volume 28 . Number 2 . 2001 . 143151

connected, flat patches joined by edges, or smooth transitional surfaces like fillets. At the edges or the transitional surfaces there usually exist big curvatures. It is a reasonable assumption that surface-based manufacturing can be carried out satisfactorily on relatively flat patches. The more flat, or less curvatured a patch is, the easier it is to find tool configurations or paths for each patch. Hence it is desirable to break up the compound surfaces into flat patches so that no big curvatures exist on each flat patch. Obviously, there is a trade-off between the number of patches and the curvature of each patch. The less curvature of each patch means more patches. Too many patches are not desirable in the sense of efficiency for tool planning. A flat patch is a collection of connected triangles, which correspond to a certain area of continuous part surface and satisfy the flat constraint: the angle  between the normal of any triangle in the patch and the average normal of the patch is within certain threshold th. Here the average normal of a patch is the area-weighted average normal of all the normals in that patch. To decompose compound surfaces into several flat patches, the following growth algorithm is proposed. (1) Select a triangle with the biggest area of all the triangles as the seed triangle. (2) Take it as the initial patch. (3) For each edge-neighboring triangle of the current patch, do (4)-(5). (4) Put the edge-neighboring triangle into the current patch and form a new patch. Calculate the average normal. (5) Test the flat constraint on the new patch. If it fails, get rid of the newly added edgeneighboring triangle. Otherwise keep the triangle and update the current patch. (6) If there is any new triangle added in the last loop, return to (3). Else (7). (7) If there is a triangle that is not in any patch, select a triangle with the biggest area among the remaining triangles as the seed triangle and repeat (2)-(7). Else Stop. The growth algorithm is illustrated in Figure 3. General approach for tool planning Figure 4 describes the general approach to solve the tool planning problem. The original part CAD model can be in any standard CAD data format such as IGES

Figure 3 Growth algorithm

Figure 4 General approach

format. First, the triangular representation of the original part surfaces is generated by 2-D triangle tessellation software. Then all the triangles are decomposed into several flat patches by setting suitable thresholds th. For each flat patch specific algorithms are developed to find configurations or paths of the tool that satisfy the given task constraints. In the following two sections, the general approach is applied on two different applications. One is vision sensor (camera) planning and the other is spray painting gun path planning. Both problems are practically important in automotive sheet metal manufacturing.

Application one: vision sensor planning

Background The structured light method, which obtains 3-D coordinates by projecting specific light patterns on the surfaces of the measured object, is one of the active part dimensional

145

CAD-guided robot motion planning

Weihua Sheng, Ning Xi, Mumin Song and Yifan Chen

Industrial Robot: An International Journal Volume 28 . Number 2 . 2001 . 143151

inspection methods (Hu and Stockman, 1989). Camera configurations, such as position, orientation, and optical settings are critical to the structured light method. They are usually planned based on human experience, thus the measurements suffer considerable human errors and low efficiency. It is highly desirable to develop a camera positioning system that can plan and realize the camera configurations in a fully automated, accurate and efficient way. Since the late 1980s, several vision sensor planning techniques have been proposed (Tarabanis et al., 1995). Basically, they fall into two categories: generate-and-test and synthesis. The generate-and-test approach simplifies the sensor planning problem into a search problem in a restricted solution space. It is straightforward and easy to implement. However, the computational cost is very high due to the large number of candidate viewpoints (Sakane et al., 1987; Sakane et al., 1992). Contrary to the generate-and-test approach, the synthesis approach forces an understanding of the causal relationships between the parameters to be planned and the goals to be achieved. Task constraints are modeled as analytic functions and the task is formulated as a constraint satisfying problem (Cowan and Kovesi,1988; Tarabanis et al., 1994, 1995). In our problem, to satisfy the requirements of the structured light method, the camera planning is based on the following four constraints: visibility, field of view, focus and resolution. A set of camera viewpoints should be found such that for any point on the surface of the part, there exists at least one viewpoint such that the point and its neighboring small area is visible, in the field of view, in focus and resolvable. Camera model and task constraints Instead of a simple pinhole model, in our system, a thick lens model used by Tarabanis et al. (1995) is adopted, which is more appropriate to model compound lens system in practical use. Based on the thick lens model, the view cone of the camera can be modeled as in Figure 5. The visibility guarantees that the line of sight is not occluded by any objects between the camera and the surface point to be inspected. The field of view constraint requires that the interested point be imaged to the active area of the image plane. Usually the

Figure 5 The camera model

field of view of a camera is represented by a field of view angle . The focus constraint guarantees that the object of interest will be sharply imaged. According to Tarabanis et al. (1994), the viewpoint region that satisfies the focus constraint for a given patch is the band between the planes perpendicular to the viewing direction and at two depth of field distances, D1 and D2 from the farthest and nearest patch vertex respectively. The resolution constraint ensures that a specified small distance be imaged to at least one pixel on the image plane. According to Cowan and Kovesi (1988), the viewpoint region satisfying resolution constraint for a given patch is the intersection of the resolution balls corresponding to each vertex on the patch. Viewpoint generation A recursive method is developed to find the camera viewpoints for a flat patch. First, a bounding box is constructed for a given flat patch by projecting all the vertices on three orthonormal directions (front, up and right) and finding the corresponding farthest and nearest project point. The three orthonormal directions are determined as follows: (1) the opposite of the average normal of the given patch is chosen as the front direction; (2) after all vertices on the patch are projected to a plane that is perpendicular to the front direction; (3) the up and right direction are determined so that the smallest rectangular field of view with specified length-width ratio can cover all the project points. Second, based on the bounding box, the viewpoints that satisfy the four constraints can be located as follows: the center of the bounding box is chosen as the look-at position

146

CAD-guided robot motion planning

Weihua Sheng, Ning Xi, Mumin Song and Yifan Chen

Industrial Robot: An International Journal Volume 28 . Number 2 . 2001 . 143151

of the camera and the up direction of the bounding box as the look-up direction. The look-from position of the camera is restricted to the line of sight which passes through the lookat point. To determine it, the line of sight is intersected by three constraint regions: the field of view region, which gives the lowest position of camera on the line of sight; the focus region, which determines two points on the line of sight corresponding to the two planes at distances D1 and D2 from the farthest and nearest patch vertex; the resolution region, which is the intersection of the resolution balls for each vertex on the patch. Any position in the intersection of the line of sight and three constraint regions is an admissible look-from position. In practice, we choose the lowest solution position to obtain a larger image. If the intersection of the line of sight and constraint regions of the current patch is null, which means no viewpoint exists that satisfies all constraints, the patch is split into two subpatches using the center plane of the bounding box along the length direction. For each subpatch, a new bounding box is constructed and the viewpoints are searched. This procedure is repeated until every patch or subpatch finds its viewpoints. Implementation The part used in the experiment is a checking gauge from Ford Motor Company. This gauge possesses most of the common features shared by automotive body designs. The deviation error of tessellation is 3mm, which results in 617 triangles. Figures 6 and 7 show the original part and the triangulation representation. A Hitachi KPD-50 CCD camera with a H0612FI lens is adopted. After running the program, the part surfaces are partitioned into 18 flat patches, which result in 23 viewpoints. The fields of
Figure 6 The original part

Figure 7 The tessellation representation

view of all the viewpoints for the checking gauge are shown in Figure 8. Experimental setup and eye-in-hand calibration The experiment setup is shown in Plate 1. The PUMA560 manipulator is controlled by a PC and a controller. The control software runs in QNX4.25 real-time operating system. Pictures taken from individual viewpoints are compared to the corresponding 3-D scenes rendered by Okino's NuGrafTM (Okino Computer Graphics Inc, 1998), a software that can import a CAD model and render it with a given camera viewpoint. Figure 9 shows two pictures and their corresponding rendering scenes. The above results show that the pictures and the 3-D rendering scenes match quite well in the sense of the field of view and the visibility. The sharpness and resolution of
Figure 8 The fields of view

147

CAD-guided robot motion planning

Weihua Sheng, Ning Xi, Mumin Song and Yifan Chen

Industrial Robot: An International Journal Volume 28 . Number 2 . 2001 . 143151

Plate 1 The experimental setup

Application two: spray painting gun path planning

Background Spray painting is an important process in automotive manufacturing. Research in automated painting path generation is aimed at eliminating the tedious and timeconsuming human teaching of robot manipulators, and generating spray painting paths which are time-efficient, satisfy coverage requirements and minimize the wastage of paint. Commercial software like ROBCAD/PaintTM (Tecnomatix, 1999) can generate painting gun paths and simulate the painting process, however, the gun paths are obtained in an interactive way between the user and ROBCAD/Paint, which has a low efficiency and is error-prone. To achieve less painting time, uniform, full paint coverage, and low wastage, both the spatial paths and the velocity of the spray gun should be planned. However, they can be solved in separate stages. Ramabhadran and Antonio reported their work in spatial path planning in (Ramabhadran and Antonio (1996) and velocity planning in (Ramabhadran and Antonio (1997)).

the interested area in pictures are also within requirements. A thorough check on all the pictures shows that the viewpoints and their realization are satisfactory, considering the positioning error of the robot and the calibration error. As another important issue, the coverage of all the surfaces by the pictures is guaranteed by the algorithm itself. More different parts are tested in our experiment and the results are summarized in Table I.

Figure 9 Pictures and rendering images

148

CAD-guided robot motion planning

Weihua Sheng, Ning Xi, Mumin Song and Yifan Chen

Industrial Robot: An International Journal Volume 28 . Number 2 . 2001 . 143151

Table I Results Part Front door Gauge Fender Pillar No. of triangle 584 617 2,602 6,011 No. of flat patch 24 18 109 116 No. of viewpoints 27 23 116 129 Running time (s) 20 22 84 180

Figure 10 Spray gun model

Usually, the velocity planning, or trajectory planning, plays an important role in achieving minimum painting time and uniform coverage, while the spatial path planning determines the paint wastage and affects the painting time and the coverage as well. Currently, only spatial path planning is our concern. Ramabhadran and Antonio (1996) provided approaches to deal with simple surfaces, for example, planes. Suh et al. (1991) developed an automatic trajectory planning system (ATPS) for spray painting robots which dealt with free-form surfaces. Their method was based on individual parametric surface. Since each parametric surface has its own variables, different from those of other surfaces, it becomes difficult to generate a unified direction for path planning (Lai and Wang, 1994). Additionally, the large number of surfaces will result in a great number of gun paths. So, as far as the painting time is concerned, their approach is not sufficient to generate satisfying gun paths for parts with compound surfaces. The challenge is to solve the spatial path planning problem, based on the part CAD model, in a global environment instead of solving it for each individual surface that is usually adopted in previous work. Thus, it is important to utilize the global geometric information of the parts in generating spraying paths. Spray gun model and task constraints A spray gun can be modeled as a spray cone. Its 2D model is shown in Figure 10. Paint is emitted from the nozzle within a small angle to the central axis z. To paint a portion of the surface, three task constraints should be satisfied: (1) For a point to receive paint, it should be within the spray cone of the gun. ! ! (2) The angle  between po and n should be less than certain value, th. This constraint guarantees that the paint can effectively accumulate on that point.

(3) The distance between the point to be painted and the spray nozzle should be within certain range, that is, within the two planes corresponding to distance D1 and D2. D = D1 D2 is called the effective range of the spray gun. To achieve uniform coverage of the paint, the paint deposition profile should be taken into consideration. Usually, it is a 2-D profile as is shown in Figure 11 to reflect the deposited thickness when the spray gun is moving in straight line at constant velocity above a plane. According to the profile, it is obvious that the distance between two path runs should be overlapped to achieve certain uniformity. Path generation The growth algorithm is applied to decompose the compound surfaces into several flat patches, such that for each flat patch, the three painting task constraints can be easily satisfied. Based on the spray gun model, a ``2-D curvature'' of the patch can be estimated, which is proportional to the angle  th and inversely proportional to the spray width W. It
Figure 11 The paint deposition profile

149

CAD-guided robot motion planning

Weihua Sheng, Ning Xi, Mumin Song and Yifan Chen

Industrial Robot: An International Journal Volume 28 . Number 2 . 2001 . 143151

can be shown that when the local 2-D curvature of a patch is less than a certain threshold, Kmax, the patch can be painted satisfactorily using straight gun paths. Since, usually, the spray width W is small and the angle th is relatively big, the 2-D curvature is relatively large. For part surfaces commonly seen in sheet metal manufacturing, a suitable flat patch threshold th can be determined such that the local 2-D curvature of each flat patch is less than Kmax. Hence, once the flat patches are formed, the spray constraints can be reduced to only one constraint, that is, to keep the spray gun a fixed distance, H, away from the surface to be painted. A spray path consists of a sequence of spray gun configurations. A frame is introduced to specify a configuration of the spray gun. This frame has a distance of H from the nozzle of the gun, o, as is shown in Figure 10. The X axis is the moving direction of the spray gun. Thus the spray paths can be represented by a sequence of frames on the surface. A ``cutting'' algorithm is used to generate gun paths for each flat patch. First, the bounding box of the flat patch is constructed; then, based on this bounding box, do the following steps: (1) Build parallel cutting planes with inbetween distance d across the whole patch, each cutting plane is parallel to the right side rectangle of the bounding box. d is smaller than W to allow some overlap and to meet certain thickness requirements as well. (2) Compute the intersection points of each cutting plane and the patch. Essentially, these points are the intersections of the cutting plane and the edges of the triangles in that patch. (3) Determine the spray gun frames: choose the opposite direction of the normal at each intersection point as the Z axis of the spray gun frame at that point; choose the moving direction of the spray gun as the X axis; find the Y axis to complete the right-hand frame. (4) Connect the frames sequentially to form a continuous path run on the patch. (5) Connect all the runs to form a complete path for the patch. By joining the path for each patch sequentially, the overall paths for the whole part are obtained.

Implementation The part used to test the algorithm is a car hood from Ford Motor Company, which is shown in Figure 12. The deviation error of tessellation is 3mm. The total number of triangles is 4,320. The parameters used in the program are:  h = 20cm; w = 5cm; d = 2.5cm, th : 5 The generated spray gun paths are shown in Figure 13 (the frames representing the orientation are not shown in this figure, for clarity). Simulation in ROBCAD To verify them, the spray gun paths are input to ROBCAD/Paint to simulate the painting process. The robot model used is a Puma560. The work-cell setup is shown in Figure 14.
Figure 12 The part to be painted

Figure 13 The generated spray gun paths

150

CAD-guided robot motion planning

Weihua Sheng, Ning Xi, Mumin Song and Yifan Chen

Industrial Robot: An International Journal Volume 28 . Number 2 . 2001 . 143151

Figure 14 ROBCAD/paint simulation

References
Cowan, C.K. and Kovesi, P.D. (1988), ``Automatic sensor placement from vision task requirements'', IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 10 No. 3, pp. 407-16. Hu, G. and Stockman, G. (1989), ``3D surface solution using structured light and constraint propagation'', IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 11 No. 4, pp. 390-402. Lai, J.Y. and Wang, D.J. (1994), ``A strategy for finish cutting path generation of compound surfaces'', Computers in Industries, Vol. 25, pp. 189-209. Marefat, M. and Kashyap, R.L. (1992), ``Automatic construction of process plans from solid model representations'', IEEE Transactions on Systems, Man, Cybernetics, Vol. 22 No. 5, pp. 1097-115. Okino Computer Graphics Inc. (1998), NuGraf Rendering System, Ontario, Canada. Ramabhadran, R. and Antonio, J.K. (1996), ``Planning spatial paths for automated spray coating applications'', in Proceedings of 1996 IEEE International Conference on Robotics and Automation, No. 2, pp. 1255-60. Ramabhadran, R. and Antonio, J.K. (1997), ``Fast solution techniques for a class of optimal trajectory planning problems with applications to automated spray coating'', IEEE Computer Graphics and Application, Vol. 13 No. 4, pp. 519-30. Sakane, S., Ishii, M. and Kakikura, M. (1987), ``Occlusion avoidance of visual sensors based on a hand eye action simulator system: HEAVEN'', Advanced Robotics, Vol. 2 No. 2, pp. 149-65. Sakane, S., Sato, T., Niepold, R. and Shirai, Y. (1992), ``Illumination setup planning for a hand-eye system based on an environmental model'', Advanced Robotics, Vol. 6 No. 4, pp. 461-82. Suh, S.H., Woo, I.K. and Noh, S.K. (1991), ``Development of an automatic trajectory planning system (ATPS) for spray painting robots'', in Proceedings of 1991 IEEE International Conference on Robotics and Automation, pp. 1948-55. Tarabanis, K.A., Allen, P.K. and Tsai, R.Y. (1995), ``A survey of sensor planning in computer vision'', IEEE Transactions on Robotics and Automation, Vol. 11 No. 1, pp. 86-104. Tarabanis, K.A., Tsai, R.Y. and Allen, P.K. (1994), ``Analytical characterization of the feature detectability constraints of resolution, focus, and field-of-view for vision sensor planning'', CVGIP: Image Understanding, Vol. 59 No. 3, pp. 340-58. Tarabanis, K.A., Tsai, R.Y. and Allen, P.K. (1995), ``The MVP sensor planning system for robotic vision tasks'', IEEE Transactions on Robotics and Automation, Vol. 11 No. 1, pp. 72-85. Tecnomatix (1999), ROBCAD/Paint Training, Michigan. Zhang, H.C. and Alting, L. (1994), Computerized Manufacturing Process Planning Systems, Chapman & Hall, London.

ROBCAD/Paint displays the coverage thickness using color scales. The resulting coverage colors show that the paint fully covers all the part surfaces with a small amount of wastage and a moderate painting time. Table II summarizes a comparison of the performances among different path generating methods, i.e. the surface-based method, the manual method in ROBCAD/ Paint and our method. It is obvious that our method is superior to the other two methods.

Conclusions
In this paper, a general framework is developed for robot motion planning in surface-based sheet metal manufacturing. The planner is based on the triangular representation of the part surfaces. A growth algorithm is proposed to decompose the compound surfaces into flat patches. For each flat patch, specific algorithms are developed to satisfy the task constraints for different applications. Simulation or implementation is carried out for each application. This general frame can be applied to various surface-based applications in manufacturing industry.
Table II Comparison summary Item Planning time No. of paths Coverage Wastage Painting time Surface-based method Short Big 100 per cent Much Long ROBCAD method Long Medium 100 per cent Much Medium Our method Short Small 100 per cent Little Short

151