RELATED APPLICATIONS
The current application claims the benefit of and takes the priority of the earlier filing dates of the following U.S. provisional application No. 62/786,469, filed 30 Dec. 2018, called ZAdvanced-6-prov, titled “System and Method for Extremely Efficient Image and Pattern Recognition and General-Artificial Intelligence Platform”. The current application is also a CIP (Continuation-in-part) of another co-pending U.S. application Ser. No. 15/919170, filed 12 Mar. 2018, called Zadeh-101-cip-cip, titled “System and Method for Extremely Efficient image and Pattern Recognition and Artificial Intelligence Platform”, which is a CIP (Continuation-in-part) of another co-pending U.S. application Ser. No. 14/218,923, filed 18 Mar. 2014, called Zadeh-101-CH, which is now issued as U.S. Pat. No. 9,916,538 on 13 Mar. 2018, which is a CIP (Continuation-in-part) of another co-pending U.S. application Ser. No. 13/781,303, filed Feb. 28, 2013, called ZAdvanced-1, now U.S. Pat. No. 8,873,813, issued on 28 Oct. 2014, which claims the benefit of and takes the priority of the earlier filing date of the following U.S. provisional application No. 61/701,789, filed Sep. 17, 2012, called ZAdvanced-1-prov. The application Ser. No. 14/218,923 also claims the benefit of and takes the priority of the earlier filing dates of the following U.S. provisional application Nos. 61/802,810, filed Mar. 18, 2013, called ZAdvanced-2-prov; and 61/832,816, filed Jun. 8, 2013, called ZAdvanced-3-prov; and 61/864,633, filed Aug. 11, 2013, called ZAdvanced-4-prov; and 61/871,860, filed Aug. 29, 2013, called ZAdvanced-5-prov. The application Ser. No. 14/218,923 is also a CIP (Continuation-in-part) of another co-pending U.S. application Ser. No. 14/201,974, filed 10 Mar. 2014, called Zadeh-101-Cont-4, now as U.S. Pat. No. 8,949,170, issued on 3 Feb. 2015, which is a Continuation of another U.S. application Ser. No. 13/953,047, filed Jul. 29, 2013, called Zadeh-101-Cont-3, now U.S. Pat. No. 8,694,459, issued on 8 Apr. 2014, which is also a Continuation of another co-pending application Ser. No. 13/621,135, filed Sep. 15, 2012, now issued as U.S. Pat. No. 8,515,890, on Aug. 20, 2013, which is also a Continuation of Ser. No. 13/621,164, filed Sep. 15, 2012, now issued as U.S. Pat. No. 8,463,735, which is a Continuation of another application, Ser. No. 13/423,758, filed Mar. 19, 2012, now issued as U.S. Pat. No. 8,311,973, which, in turn, claims the benefit of the U.S. provisional application No. 61/538,824, filed on Sep. 24, 2011. The current application incorporates by reference all of the applications and patents/provisionals mentioned above, including all their Appendices and attachments (Packages), and it claims benefits to and takes the priority of the earlier filing dates of all the provisional and utility applications or patents mentioned above. Please note that most of the Appendices and attachments (Packages) to the specifications for the above-mentioned applications and patents (such as U.S. Pat. No. 8,311,973) are available for public view, e.g., through Public Pair system at the USPTO web site (www.uspto.gov), with some of their listings given below in the next section:
ATTACHED PACKAGES AND APPENDICES TO PRIOR SPECIFICATIONS (e.g., U.S. Pat. No. 8,311,973 AND Zadeh401-CIP)
(All incorporated by reference, herein, in the current application.)
In addition to the provisional cases above, the teachings of all 33 packages (the PDF files, named “Packages 1-33”) attached with some of the parent cases' filings (as Appendices) (such as U.S. Pat. No. 8,311,973 (i.e., Zadeh-101 docket)) are incorporated herein by reference to this current disclosure.
Furthermore, “Appendices 1-5” of Zadeh-101-CIP (i.e., Ser. No. 14/218,923) are incorporated herein by reference to this current disclosure.
To reduce the size of the appendices/disclosure, these Packages (Packages 1-33) and Appendices (Appendices 1-5) are not repeated here again, but they may be referred to/incorporated in, in the future from time to time in the current or the children/related applications, both in spec or claims, as our own previous teachings.
However, the new Appendices attached to this current application is now numbered after the appendices mentioned above, i.e., starting with Appendix 6, for this current application, to make it easier to refer to them in the future.
Please note that Appendices 1-5 (of Zadeh-101-CIP (i.e., Ser. No. 14/218,923)) are identified as:
-
- Appendix 1: article about “Approximate Z-Number Evaluation based on Categorical Sets of Probability Distributions” (11 pages)
- Appendix 2: hand-written technical notes, formulations, algorithms, and derivations (5 pages)
- Appendix 3: presentation about “Approximate Z-Number Evaluation Based on Categorical Sets of Probability Distributions” (30 pages)
- Appendix 4: presentation with FIGS. from B1 to B19 (19 pages)
- Appendix 5: presentation about “SVM Classifier” (22 pages)
Please note that Appendices 6-10 (of Zadeh-101-CIP-CIP (i.e., the current application)) are identified as:
-
- Appendix 6: article/journal/technical/research/paper about “The Information Principle”, by Prof. Lotfi Zadeh, Information Sciences, submitted 16 May 2014, published 2015 (10 pages)
- Appendix 7: presentation/conference/talk/invited/keynote speaker/lecture about “Stratification, target set reachability, and incremental enlargement principle”, by Prof. Lotfi Zadeh, UC Berkeley, World Conference on Soft Computing, May 22, 2016 (14 pages, each page including 9 slides, for a total of 126 slides) (first version prepared on Feb. 8, 2016)
- Appendix 8: article about “Stratification, quantization, target set reachability, and incremental enlargement principle”, by Prof. Lotfi Zadeh, for Information Sciences, received 4 Jul. 2016 (17 pages) (first version prepared on Feb. 5, 2016)
- Appendix 9: This shows the usage of visual search terms for our image search engine (1 page), which is the first in the industry. It shows an example for shoes (component or parts matching, from various shoes), using ZAC/our technology and platform. For example, it shows the search for: “side look like shoe number 1, heel look like shoe number 2, and toe look like shoe number 3”, based on what the user is looking/searching for. In general, we can have a combination of conditions, e.g.: (R1 AND R2 AND . . . AND Rn), or any logical search terms or combinations or operators, e.g., [R1 OR (R2 AND R3)], which is very helpful for e-commerce or websites/e-stores.
- Appendix 10: “Brief Introduction to AI and Machine Learning”, for conventional tools and methods, sometimes used or referred to in this invention, for completeness and as support of the main invention, or just for the purpose of comparison with the conventional tools and methods.
Please note that Appendices 11-13 (of ZAdvanced-6-prov) are identified as:
-
- Appendix 11 “ZAC General-AI Platform for 3D Object Recognition & Search from any Direction (Revolutionary Image Recognition & Search Platform)”, for descriptions and details of General-AI Platform, which includes Explainable-AI (or XAI or X-AI or Explainable-Artificial Intelligence), as well. This also describes ZAC features and advantages over NN (or CNN or Deep CNN or Deep Convolutional Neural Net or ResNet). This also describes applications, markets, and use cases/examples/embodiments for ZAC tech/algorithms/platform.
- Appendix 12: ZAC platform and operation, with features, architecture, modules, layers, and components. This also describes ZAC features and advantages over NN (or CNN or Deep CNN or Deep Convolutional Neural Net or ResNet),
- Appendix 13: Some examples/embodiments/tech descriptions for ZAC tech/platform (General-AI Platform).
Please note that Appendix 14 (of Zadeh-101-cip-cip-cip) (i.e., the current application) is identified as ZAC Explainable-AI, which is a component of ZAC General-AI Platform. This also describes applications, markets, and use cases/examples/embodiments for ZAC tech/algorithms/platform. This also describes ZAC features and advantages over NN (or CNN or Deep CNN or Deep Convolutional Neural Net or ResNet).
Please note that Packages 1-33 (of U.S. Pat. No. 8,311,973) are also one of the inventor's (Prof. Lotfi Zadeh's) own previous technical teachings, and thus, they may be referred to (from time-to-time) for further details or explanations, by the reader, if needed.
Please note that Packages 1-25 had already been submitted (and filed) with our provisional application for one of the parent cases.
Packages 1-12 and 15-22 are marked accordingly at the bottom of each page or slide (as the identification). The other Packages (Packages 13-14 and 23-33) are identified here:
-
- Package 13: 1 page, with 3 slides, starting with “
FIG. 1 . Membership function of A and probability density function of X” - Package 14: 1 page, with 5 slides, starting with “
FIG. 1 . f-transformation and f-geometry. Note that fuzzy figures, as shown, are not hand drawn. They should be visualized as hand drawn figures.” - Package 23: 2-page text, titled “The Concept of a Z-number a New Direction in Computation, Lotfi A. Zadeh, Abstract” (dated Mar. 28, 2011)
- Package 24: 2-page text, titled “Prof. Lotfi Zadeh, The Z-mouse—a visual means of entry and retrieval of fuzzy data”
- Package 25: 12-page article, titled “Toward Extended Fuzzy Logic A First Step, Abstract”
- Package 26: 2-page text, titled “Can mathematics deal with computational problems which are stated in a natural language?, Lotfi A. Zadeh, Sep. 30, 2011, Abstract” (Abstract dated Sep. 30, 2011)
- Package 27: 15 pages, with 131 slides, titled “Can Mathematics Deal with Computational Problems Which are Stated in a Natural Language?, Lotfi A. Zadeh” (dated Feb. 2, 2012)
- Package 28: 14 pages, with 123 slides, titled “Can Mathematics Deal with Computational Problems Which are Stated in a Natural Language?, Lotfi A. Zadeh” (dated Oct. 6, 2011)
- Package 29: 33 pages, with 289 slides, titled “Computing with Words Principal Concepts and Ideas, Lotfi A. Zadeh” (dated Jan. 9, 2012)
- Package 30: 23 pages, with 205 slides, titled “Computing with Words Principal Concepts and Ideas, Lotfi A. Zadeh” (dated May 10, 2011)
- Package 31: 3 pages, with 25 slides, titled “Computing with Words Principal Concepts and Ideas, Lotfi A. Zadeh” (dated Nov. 29, 2011)
- Package 32: 9 pages, with 73 slides, titled “Z-NUMBERS—A NEW DIRECTION IN THE ANALYSIS OF UNCERTAIN AND IMPRECISE SYSTEMS, Lotfi A. Zadeh” (dated Jan. 20, 2012)
- Package 33: 15 pages, with 131 slides, titled “PRECISIATION OF MEANING—A KEY TO SEMANTIC COMPUTING, Lotfi A, Zadeh” (dated Jul. 22, 2011)
- Package 13: 1 page, with 3 slides, starting with “
Please note that all the Packages and Appendices (prepared by one or more of the inventors here) were also identified by their PDF file names, as they were submitted to the USPTO electronically.
BACKGROUND OF THE INVENTION
Professor Lotfi A. Zadeh, one of the inventors of the current disclosure and some of the parent cases, is the “Father of Fuzzy Logic”. He first introduced the concept of Fuzzy Set and Fuzzy Theory in his famous paper, in 1965 (as a professor of University of California, at Berkeley). Since then, many people have worked on the Fuzzy Logic technology and science. Dr. Zadeh has also developed many other concepts related to Fuzzy Logic. He has invented Computation-with-Words (CWW or CW), e.g., for natural language processing (NLP) and analysis, as well as semantics of natural languages and computational theory of perceptions, for many diverse applications, which we address here, as well, as some of our new/innovative methods and systems are built based on those concepts/theories, as their novel/advanced extensions/additions/versions/extractions/branches/fields. One of his last revolutionary inventions is called Z-numbers, named after him (“Z” from Zadeh), which is one of the many subjects of the (many) current inventions. That is, some of the many embodiments of the current inventions are based on or related to Z-numbers. The concept of Z-numbers was first published in a recent paper, by Dr. Zadeh, called “A Note on Z-Numbers”, Information Sciences 181 (2011) 2923-2932.
However, in addition, there are many other embodiments in the current disclosure that deal with other important and innovative topics/subjects, e.g., related to General AI, versus Specific or Vertical or Narrow AI, machine learning, using/requiring only a small number of training samples (same as humans can do), learning one concept and use it in another context or environment (same as humans can do), addition of reasoning and cognitive layers to the learning module (same as humans can do), continuous learning and updating the learning machine continuously (same as humans can do), simultaneous learning and recognition (at the same time) (same as humans can do), and conflict and contradiction resolution (same as humans can do), with application, e.g., for image recognition, application for any pattern recognition, e.g., sound or voice, application for autonomous or driverless cars, application for security and biometrics, e.g., partial or covered or tilted or rotated face recognition, or emotion and feeling detections, application for playing games or strategic scenarios, application for fraud detection or verification/validation, e.g., for banking or cryptocurrency or tracking fund or certificates, application for medical imaging and medical diagnosis and medical procedures and drug developments and genetics, application for control systems and robotics, application for prediction, forecasting, and risk analysis, e.g., for weather forecasting, economy, oil price, interest rate, stock price, insurance premium, and social unrest indicators/parameters, and the like,
In the real world, uncertainty is a pervasive phenomenon. Much of the information on which decisions are based is uncertain. Humans have a remarkable capability to make rational decisions based on information which is uncertain, imprecise and/or incomplete. Formalization of this capability is one of the goals of these current inventions, in one embodiment.
Here are some of the publications on the related subjects, for some embodiments:
[1] R., Ash, Basic Probability Theory, Dover Publications, 2008.
[2] J-C. Buisson, Nutri-Educ, a nutrition software application for balancing meals, using fuzzy arithmetic and heuristic search algorithms, Artificial Intelligence in Medicine 42, (3), (2008) 213-227.
[3] E. Trillas, C. Moraga, S. Guadarrama, S. Cubillo and E. Castiñeira, Computing with Antonyms, In: M. Nikravesh, J. Kacprzyk and L. A. Zadeh (Eds.), Forging New Frontiers: Fuzzy Pioneers I, Studies in Fuzziness and Soft Computing Vol 217, Springer-Verlag, Berlin Heidelberg 2007, pp. 133-153.
[4] R. R. Yager, On measures of specificity, In: O. Kaynak, L. A. Zadeh, B. Turksen, I. J. Rudas (Eds.), Computational Intelligence: Soft Computing and Fuzzy-Neuro :Integration with Applications, Springer-Verlag, Berlin, 1998, pp. 94-113.
[5] L. A. Zadeh, Calculus of fuzzy restrictions, In: L. A. Zadeh, K. S. Fu, K. Tanaka, and M. Shimura (Eds.), Fuzzy sets and Their Applications to Cognitive and Decision Processes, Academic Press, New York, 1975, pp. 1-39.
[6] L. A. Zadeh, The concept of a linguistic variable and its application to approximate reasoning,
Part Information Sciences 8 (1975) 199-249;
Part II: Information Sciences 8 (1975) 301-357;
Part III: Information Sciences 9 (1975) 43-80.
[7] L. A. Zadeh, Fuzzy logic and the calculi of fuzzy rules and fuzzy graphs, Multiple-Valued Logic 1, (1996) 1-38.
[8] L. A. Zadeh, From computing with numbers to computing with words—from manipulation of measurements to manipulation of perceptions, IEEE Transactions on Circuits and Systems 45, (1999) 105-119.
[9] L. A. Zadeh, The Z-mouse a visual means of entry and retrieval of fuzzy data, posted on BISC Forum, Jul. 30, 2010. A more detailed description may be found in Computing with Words—principal concepts and ideas, Colloquium PowerPoint presentation, University of Southern California, Los Angeles, Calif., Oct. 22, 2010.
As one of the applications mentioned here in this disclosure, for comparisons, some of the search engines or question-answering engines in the market (in the recent years) are (or were): Google®, Yahoo®, Autonomy, M®, Fast Search, Powerset® (by Xerox® PARC and bought by Microsoft®), Microsoft® Bing, Wolfram®, AskJeeves, Collarity, Endeca®, Media River, Hakia®, Ask.com®, AltaVista, Excite, Go Network, HotBot®, Lycos®, Northern Light, and Like.com.
Other references on some of the related subjects are:
[1] A. R. Aronson, B. E. Jacobs, J. Minker, A note on fuzzy deduction, J. ACM27 (4) (1980), 599-603.
[2] A. Bardossy, L. Duckstein, Fuzzy Rule-based Modelling with Application to Geophysical, Biological and Engineering Systems, CRC Press, 1995.
[3] T. Berners-Lee, J. Hendler, Q. Lassila, The semantic web, Scientific American 284 (5) (2001), 34-43.
[4] S. Brin, L. Page, The anatomy of a large-scale hypertextual web search engine, Computer Networks 30 (1-7) (1998), 107-117.
[5] W. J. H. J. Bronnenberg, M. C. Bunt, S. P. J. Lendsbergen, R. H. J. Scha,W. J. Schoenmakers, E. P. C., van Utteren, The question answering system PHLIQA1, in: L. Bola (Ed.), Natural Language Question Answering Systems, Macmillan, 1980.
[6] L. S. Coles, Techniques for information retrieval using an inferential question-answering system with natural language input, SRI Report, 1972.
[7] A. Di Nola, S. Sessa, W. Pedrycz, W. Pei-Zhuang, Fuzzy relation equation under a class of triangular norms: a survey and new results, in: Fuzzy Sets for Intelligent Systems, Morgan Kaufmann Publishers, San Mateo, Calif., 1993, pp. 166-189.
[8] A. Di. Nola, S. Sessa, W. Pedrycz, E. Sanchez, Fuzzy Relation Equations and their Applications to Knowledge Engineering, Kluwer Academic Publishers, Dordrecht, 1989.
[9] D. Dubois, H. Prade, Gradual inference rules in approximate reasoning, Inform. Sci. 61 (1-2) (1992), 103-122.
[10] D. Filev, R. R. Yager, Essentials of Fuzzy Modeling and Control, Wiley-Interscience, 1994.
[11] J. A. Goguen, The logic of inexact concepts, Synthese 19 (1969), 325-373.
[12] M. Jamshidi, A. Titli, L. A. Zadeh, S. Boverie (Eds.), Applications of Fuzzy Logic—Towards High Machine intelligence Quotient Systems, Environmental and Intelligent Manufacturing Systems Series, vol. 9, Prentice-Hall, Upper Saddle River, N.J., 1997.
[13] A. Kaufmann, M. M. Gupta, Introduction to Fuzzy Arithmetic: Theory and Applications, Van Nostrand. New York, 1985.
[14] D. B. Lenat, CYC: a large-scale investment in knowledge infrastructure, Comm.ACM38 (11) (1995), 32-38.
[15] E. H. Mamdani, S. Assilian, An experiment in linguistic synthesis with a fuzzy logic controller, Int. J. Man—Machine Studies 7 (1975), 1-13.
[16] J. R. McSkimin, Minker, The use of a semantic network in a deductive question-answering system, in: IJCAI, 1977, pp. 50-58.
[17] R. E. Moore, Interval Analysis, SIAM Studies in Applied Mathematics, vol. 2, Philadelphia, Pa., 1979.
[18] M. Nagao, J. Tsujii, Mechanism of deduction in a question-answering system with natural language input, in: ICJAI, 1973, pp. 285-290.
[19] B. H. Partee (Ed.), Montague Grammar, Academic Press, New York, 1976.
[20] W. Pedrycz, F. Gomide, Introduction to Fuzzy Sets, MIT Press, Cambridge, Mass., 1998.
[21] F. Rossi, P. Codognet (Eds.), Soft Constraints, Special issue on Constraints, vol. 8, N. 1, Kluwer Academic Publishers, 2003.
[22] G. Shafer, A Mathematical Theory of Evidence, Princeton University Press, Princeton, N.J., 1976.
[23] M. K. Smith, C. Welty, D. McGuinness (Eds. OWL Web Ontology Language Guide, W3C Working Draft 31, 2003.
[24] L. A. Zadeh, Fuzzy sets, Inform and Control 8 (1965), 338-353.
[25] L. A. Zadeh, Probability measures of fuzzy events, J. Math. Anal. Appi. 23 (1968), 421-427.
[26] L. A. Zadeh, Outline of a new approach to the analysis of complex systems and decision processes, IEEE Trans. on Systems Man Cybemet. 3 (1973), 28-44.
[27] L. A. Zadeh, On the analysis of large scale systems, in: H. Gottinger (Ed.), Systems Approaches and Environment Problems, Vandenhoeck and Ruprecht, Gottingen, 1974, pp. 23-37.
[28] L. A., Zadeh, The concept of a linguistic variable and its application to approximate reasoning, Part I, Inform. Sci. 8 (1975), 199-249; Part II, Inform. Sci. 8 (1975), 301-357; Part Inform. Sci. 9 (1975), 43-80.
[29] L. A. Zadeh, Fuzzy sets and information granularity, in: M. Gupta, R. Ragade, R. Yager (Eds.), Advances in Fuzzy Set Theory and Applications, North-Holland Publishing Co, Amsterdam, 1979, pp. 3-18,
[30] L. A. Zadeh, A theory of approximate reasoning, in: J. Hayes, D. Michie, L. I. Mikulich (Eds.), Machine Intelligence, vol. 9, Halstead Press, New York, 1979, pp. 149-194.
[31] L. A. Zadeh, Test-score semantics for natural languages and meaning representation via PRUF, in: B. Rieger (Ed.), Empirical Semantics, Brockmeyer, Bochum, W. Germany, 1982, pp. 281-349. Also Technical Memorandum 246, AI Center, SRI International, Menlo Park, Calif., 1981.
[32] L. A. Zadeh, A computational approach to fuzzy quantifiers in natural languages, Computers and Mathematics 9 (1983), 149-184.
[33] L. A. Zadeh, A fuzzy-set-theoretic approach to the compositionality of meaning: propositions, dispositions and canonical forms, J. Semantics 3 (1983), 253-272,
[34] L. A. Zadeh, Precisiation of meaning via translation into PRUF, in: L. Vaina, J. Hintikka (Eds.), Cognitive Constraints on Communication, Reidel, Dordrecht, 1984, pp. 373-402.
[35] L. A. Zadeh, Outline of a computational approach to meaning and knowledge representation based on a concept of a generalized assignment statement, in: M. Thoma, A. Wyner (Eds.), Proceedings of the International Seminar on Artificial Intelligence and Man-Machine Systems, Springer-Verlag, Heidelberg, 1986, pp. 198-211.
[36] L. A. Zadeh, Fuzzy logic and the calculi of fuzzy rules and fuzzy graphs, Multiple-Valued Logic 1 (1996), 1-38.
[37] LA, Zadeh, Toward a theory of fuzzy information granulation and its centrality in human reasoning and fuzzy logic, Fuzzy Sets and Systems 90 (1997), 111-127.
[38] L. A. Zadeh, From computing with numbers to computing with words—from manipulation of measurements to manipulation of perceptions, IEEE Trans. on Circuits and Systems 45 (1) (1999), 105-119.
[39] L. A., Zadeh, Toward a perception-based theory of probabilistic reasoning with probabilities, J. Statist. Plann. Inference 105 (2002), 233-264.
[40] L. A. Zadeh, Precisiated natural language (PNL', AI Ntagazine 25 (3) (2004), 74-91.
[41] L. A., Zadeh, A note on web intelligence, world knowledge and fuzzy logic, Data and Knowledge Engineering 50 (2004), 291-304.
[42] L. A. Zadeh, Toward a generalized theory of uncertainty (GTU)—an outline, Inform. Sci. 172 (2005), 1-40.
[43] J. Arjona, R. Corchuelo, J. Pena, D. Ruiz, Coping with web knowledge, in: Advances in Web Intelligence, Springer-Verlag, Berlin, 2003, pp. 165-178.
[44] A. Bargiela, W. Pedrycz, Granular Computing—An Introduction, Kluwer Academic Publishers, Boston, 2003.
[45] Z. Bubnicki, Analysis and Decision Making in Uncertain Systems, Springer-Verlag, 2004.
[46] P. P. Chen, Entity-relationship Approach to Information Modeling and Analysis, North-Holland, 1983.
[47] M. Craven, D. DiPasquo, D. Freitag, A. McCallum, T. Mitchell, K. Nigam, S. Slattery, Learning to construct knowledge bases from the world wide web, Artificial Intelligence 118 (1-2) (2000), 69-113,
[48] M. J. Cresswell, Logic and Languages, Methuen, London, UK, 1973.
[49] D. Dubois, H. Prade, On the use of aggregation operations in information fusion processes, Fuzzy Sets and Systems 142 (1) (2004), 143-161.
[50] T. F. Gamat, Language, Logic and Linguistics, University of Chicago Press, 1996.
[51] M. Mares, Computation over Fuzzy Quantities, CRC, Boca Raton, Fla., 1994.
[52] V. Novak, I. Perfilieva, J. Mockor, Mathematical Principles of Fuzzy Logic, Kluwer Academic Publishers, Boston, 1999.
[53] V. Novak, I. Perfilieva (Eds.), Discovering the World with Fuzzy Logic, Studies in Fuzziness and Soft Computing, Physica-Verlag, Heidelberg, 2000.
[54] Z. Pawlak, Rough Sets: Theoretical Aspects of Reasoning about Data, Kluwer Academic Publishers, Dordrecht, 1991.
[55] M. K. Smith, C. Welty, What is ontology? Ontology: towards a new synthesis, in: Proceedings of the Second International Conference on Formal Ontology in information Systems, 2002.
However, none of the prior art teaches the features mentioned in our invention disclosure.
There are a lot of research going on today, focusing on the search engine, analytics, Big Data processing, natural language processing, economy forecasting, dealing with reliability and certainty, medical diagnosis, pattern recognition, object recognition, biometrics, security analysis, risk analysis, fraud detection, satellite image analysis, machine generated data, machine learning, training samples, and the like.
For example, see the article by Technology Review, published by MIT, “Digging deeper in search web”, Jan. 29, 2009, by Kate Greene, or search engine by GOOGLE®, MICROSOFT® (BING®), or YAHOO®, or APPLE® SIRI, or WOLFRAM® ALPHA computational knowledge engine, or AMAZON engine, or FACEBOOK® engine, or ORACLE® database, or YANDEX® search engine in Russia, or PICASA® (GOOGLE®) web albums, or YOUTUBE® (GOGGLE®) engine, or ALIBABA (Chinese supplier connection), or SPLUNK® (for Big Data), or MICROSTRATEGY® (for business intelligence), or QUID (or KAGGLE, ZESTFINANCE, APIXIO, DATAMEER, BLUEKAI, GNIP, RETAILNEXT, or RECOMMIND) (for Big Data), or paper by Viola-Jones, Viola et al., at Conference on Computer Vision and Pattern Recognition, 2001, titled “Rapid object detection using a boosted cascade of simple features”, from Mitsubishi and Compaq research labs, or paper by Alex Pentland et al., February 2000, at Computer, IFEE, titled “Face recognition for smart environments”, or GOOGLE® official blog publication, May 16, 2012, titled “Introducing the knowledge graph: things, not strings”, or the article by Technology Review, published by MIT, “The future of search”, Jul. 16, 2007, by Kate Greene, or the article by Technology Review, published by MIT, “Microsoft searches for group advantage”, Jan. 30, 2009, by Robert Lemos, or the article by Technology Review, published by MIT, “WOLFRAM ALPHA and GOOGLE face off”, May 5, 2009, by David Talbot, or the paper by Devarakonda et al., at International Journal of Software Engineering (IJSE), Vol. 2, Issue 1, 2011, titled “Next generation search engines for information retrieval”, or paper by Nair-Hinton, titled “Implicit mixtures of restricted Boltzmann machines”, NIPS, pp. 1145-1152, 2009, or paper by Nair, V. and Hinton, G. E., titled “3-D Object recognition with deep belief nets”, published in Advances in Neural information Processing Systems 22, (Y. Bengio, D. Schuurmans, Lafferty, C. K. I. Williams, and A. Culotta (Eds.)), pp 1339-1347. Other research groups include those headed by Andrew Ng, Yoshua Bengio, Fei Fei Li, Ashutosh Saxena, LeCun, Michael I. Jordan, Zoubin Ghahramani, and others in companies and universities around the world.
However, none of the prior art teaches the features mentioned in our invention disclosure, even in combination.
SUMMARY OF THE INVENTION
For one embodiment: Decisions are based on information. To be useful, information must be reliable. Basically, the concept of a Z-number relates to the issue of reliability of information. A Z-number, Z, has two components, Z=(A,B). The first component, A, is a restriction (constraint) on the values which a real-valued uncertain variable, X, is allowed to take. The second component, B, is a measure of reliability (certainty) of the first component. Typically, A and B are described in a natural language. Example: (about 45 minutes, very sure). An important issue relates to computation with Z-numbers. Examples are: What is the sum of (about 45 minutes, very sure) and (about 30 minutes, sure)? What is the square root of (approximately 100, likely)? Computation with Z-numbers falls within the province of Computing with Words (CW or CWW). In this disclosure, the concept of a Z-number is introduced and methods of computation with Z-numbers are shown. The concept of a Z-number has many applications, especially in the realms of economics, decision analysis, risk assessment, prediction, anticipation, rule-based characterization of imprecise functions and relations, and biomedicine. Different methods, applications, and systems are discussed. Other Fuzzy inventions and concepts are also discussed. Many non-Fuzzy-related inventions and concepts are also discussed.
For other embodiments: Specification also covers new algorithms, methods, and systems for artificial intelligence, soft computing, and deep/detailed learning/recognition, e.g., image recognition (e.g., for action, gesture, emotion, expression, biometrics, fingerprint, facial, OCR (text), background, relationship, position, pattern, and object), large number of images (“Big Data”) analytics, machine learning, training schemes, crowd-sourcing (using experts or humans), feature space, clustering, classification, similarity measures, optimization, search engine, ranking, question-answering system, soft (fuzzy or unsharp) boundaries/impreciseness/ambiguities/fuzziness in language, Natural Language Processing (NLP), Computing-with-Words (CWW), parsing, machine translation, sound and speech recognition, video search and analysis (e.g., tracking), image annotation, geometrical abstraction, image correction, semantic web, context analysis, data reliability (e.g., using Z-number (e.g., “About 45 minutes; Very sure”)), rules engine, control system, autonomous vehicle (e.g., self-parking), self-diagnosis and self-repair robots, system diagnosis, medical diagnosis, biomedicine, data mining, event prediction, financial forecasting, economics, risk assessment, e-mail management, database management, indexing and join operation, memory management, and data compression.
Other topics/inventions covered are, e.g.:
-
- Method and System for Identification or Verification for an Object, a Person, or their Attributes
- System and Method for Image Recognition and Matching for Targeted Advertisem*nt
- System and Method for Analyzing Ambiguities in Language for Natural Language Processing
- Application of Z-Webs and Z-factors to Analytics, Search Engine, Learning, Recognition, Natural Language, and Other Utilities
- Method and System for Approximate Z-Number Evaluation based on Categorical Sets of Probability Distributions
- Image and Video Recognition and Application to Social Network and Image and Video Repositories
- System and Method for Image Recognition for Event-Centric Social Networks
- System and Method for image Recognition for Image Ad Network
- System and Method for Increasing Efficiency of Support Vector Machine Classifiers
Other topics/inventions covered are, e.g.:
-
- a Information Principle
- Stratification
- Incremental Enlargement Principle
- Deep/detailed Machine Learning and training schemes
- Image recognition (e.g., for action, gesture, emotion, expression, biometrics, fingerprint, facial (e.g., using eigenface), monument and landmark, OCR, background, partial object, relationship, position, pattern, texture, and object)
- Basis functions
- Image and video auto-annotation
- Focus window
- Modified/Enhanced Boltzmann Machines
- Feature space translation
- Geometrical abstraction
- Image correction
- Semantic web
- Context analysis
- Data reliability
- Correlation layer
- Clustering
- Classification
- Support Vector Machines
- Similarity measures
- Optimization
- Z-number
- Z-factor
- Z-web
- Rules engine
- Control system
- Robotics
- Search engine
- Ranking
- Question-answering system
- Soft boundaries & Fuzziness in language
- Natural Language Processing (NLP)
- System diagnosis
- Medical diagnosis
- Big Data analytics
- Event prediction
- Financial forecasting
- Computing with Words (CWW)
- Parsing
- Soft boundaries & Fuzziness in clustering & classification
- Soft boundaries & Fuzziness in recognition
- Machine translation
- Risk assessment
- e-mail management
- Database management
- Indexing and join operation
- Memory management
- Sound and speech recognition
- Video search & analysis (e.g., tracking)
- Data compression
- Crowd sourcing (e.g., with experts or SMEs)
- Event-centric social networking (based on image)
- Energy
- Transportation
- Distribution of materials
- Optimization
- Scheduling
We have also introduced the first Image Ad Network, powered by our next generation image search engine.
We have introduced our novel “ZAC™ Image Recognition Platform”, which applies learning based on General-AI algorithms. This way, we need much smaller number of training samples to train (the same as humans do), e.g., for evaluating or analyzing a 3-D object/image, e.g., a complex object, such as a shoe, from any direction or angle. To our knowledge, nobody has solved this problem, yet. This is the “Holy Grail” of image recognition. Having/requiring much smaller number of training samples to train is also the “Holy Grail” of AI and machine learning. So, here, we have achieved 2 major scientific and technical milestones/breakthroughs that others have failed to obtain. (These results had been originally reported in our parent cases, as well.)
In addition, to our knowledge, this is the first successful example of application of General-AI algorithms, systems, and methods in any field, application, industry, university, research, paper, experiment, demo, or usage.
With other methods in the industry/universities, e.g., Deep Learning or Convolutional Neural
Networks or Deep Reinforcement Learning (maximizing a cumulative reward function) or variations of Neural Networks (e.g., Capsule Networks, recently introduced by Prof. Hinton, Sara Sabour, and Nicholas Frosst, from Google and U. of Toronto), these cannot be done at all, even with much larger number of training samples and much larger CPU/GPU computing time/power and much longer training time periods.
So, we have a significant advantage over the other methods in the industry/universities, as these tasks cannot be done by other methods at all.
Even for the conventional/much easier/very specific tasks, where the other AI methods are applicable/useful, we still have a huge advantage over them, by some orders of magnitude, in terms of cost, efficiency, size, training time, computing/resource requirements, battery lifetime, flexibility, and detection/recognition/prediction accuracy.
These shortcomings/failures/limitations of the other methods/systems/algorithms/results in the AI/machine learning industry/universities have been expressed/confirmed by various AI/machine learning people/researchers. For example, Prof. Hinton, a Google Fellow and a pioneer in AI from U. of Toronto, in an interview ( GIGAOM, Jan. 16, 2017), stated that, “One problem we still haven't solved is getting neural nets to generalize well from small amounts of data, and I suspect that this may require radical changes in the types of neuron we use”. In addition, in another interview (Axios, Sep. 15, 2017), he strongly cast doubts about AI's current methodologies, and said that, “My view is throw it all away and start again” Similarly, Mr. Suleyman (the head of Applied AI, now at DeepMind/Google) stated in an interview at TechCrunch (Dec. 5, 2016) that he thinks that the “general AI is still a long way off”.
So, to our knowledge, beyond the futuristic movies, wish-lists, science fiction novels, and generic non-scientific or non-technical articles (which have no basis/reliance/foundation on theory or experiment or proper/complete teachings), nobody has been successful in the application/usage/demonstration of General-AI, yet, in the AI industry or academia around the world. Thus, our demo/ZAC General-AI Image Recognition Software Platform here is a very significant breakthrough in the field/science of AI and machine learning technology. (These results had been originally reported in our parent cases, as well.)
Please note that General-AI is also called/referred to as General Artificial Intelligence (GAI), or Artificial General Intelligence (AGI), or General-Purpose AI, or Strong Artificial Intelligence (AI), or True AI, or as we call it, Thinking-AI, or Reasoning-AI, or Cognition-AI, or Flexible-AI, or Full-Coverage-AI, or Comprehensive-AI, which can perform tasks that was never specifically trained for, e.g., in different context/environment, to recycle/re-use the experience and knowledge, using reasoning and cognition layers, usually in a completely different or unexpected or very new situation/condition/environment (same as what a human can do). Accordingly, we have shown here in this disclosure a new/novel/revolutionary architecture, system, method, algorithm, theory, and technique, to implement General-AI, e.g., for 3-D image/object recognition from any directions and other applications discussed here.
Our technology here (based on General-AI) is in contrast to (versus) Specific AI (or Vertical or Functional or Narrow or Weak AI) (or as we have coined the phrase, “Dumb-AI”), because, e.g., a Specific AI machine trained for face recognition cannot do any other tasks, e.g., finger-print recognition or medical imaging recognition. That is, the Specific AI machine cannot carry over/learn from any experience or knowledge that it has gained from one domain (face recognition) into another/new domain (finger-print or medical imaging), which it has not seen before (or was not trained for before). So, Specific AI has a very limited scope/“intelligence”/functionality/usage/re-usability/flexibility/usefulness.
Please note that the conventional/current state-of-the-art technologies in the industry/academia (e.g., Convolutional Neural Nets or Deep Learning) are based on the Specific AI, which has some major/serious theoretical/practical limits. For example, it cannot perform a 3-D image/object recognition from all directions, or cannot carry over/learn from any experience or knowledge in another domain, or requires extremely large number of training samples (which may not be available at all, or is impractical, or is too expensive, or takes too long to gather or train), or requires extremely large neural network (which cannot converge in the training stage, due to too much degree of freedom, or tends to memorize (rather than learn) the patterns (which is not good for out-of-sample recognition accuracy)), or requires extremely large computing power (which is impractical, or is too expensive, or is not available, or still cannot converge in the training stage). So, they have serious theoretical/practical limitations.
In addition, in Specific AI, if a new class of objects is added/introduced/found to the universe of all objects (e.g., a new animal/species is discovered), the training has to be done from scratch. Otherwise, training on just the last object will bias the whole learning machine, which is not good/accurate for recognition later on. Thus, all weights/biases or parameters in the learning machine must be erased completely, and the whole learning, with the new class added/mixed randomly with previous ones, must be repeated again from scratch, with all parameters erased and re-done/calculated again. So, the solution is not cumulative, or scalable, or practical, at all, e.g., for daily learning or continuous learning, as is the case for most practical situations, or as how the humans or most animals do/learn/recognize. So, they have serious theoretical/practical limitations.
Furthermore, for Specific AI, the learning phase cannot be mixed with the training phase. That is, they are not simultaneous, in the same period of time. So, during the training phase, the machine is useless or idle for all practical purposes, as it cannot recognize anything properly at that time. This is not how humans learn/recognize on a daily basis. So, they have serious theoretical/practical limitations.
General-AI solves/overcomes all of the above problems, as shown/discussed here in this disclosure. So, it has a huge advantage, for many reasons, as stated here, over Specific-AI.
It is also noteworthy that using smaller CPU/GPU power enables easier integration in mobile devices and wearables and loT and telephones and watches, as an example, which, otherwise, drains the battery very quickly, and thus, requires much bigger battery or frequent recharging, which is not practical for most situations at all.
The industries/applications for our inventions are, e.g.:
-
- a Mobile devices (e.g., phones, wearable devices, eyeglasses, tablets)
- Smart devices & connected/Internet appliances
- The Internet of Things (IoT), as the network of physical devices, vehicles, home appliances, wearables, mobile devices, stationary devices, wireless or cellular devices, BlueTooth or WiFi devices, and the like, embedded with electronics, software, sensors, actuators, mechanical parts, switches, and/or connectivity, which enables these objects to connect and exchange data/commands/info/trigger events.
- Natural Language Processing
- Photo albums & web sites containing pictures
- Video libraries & web sites
- Image and video search & summarization & directory & archiving & storage
- Image & video Big Data analytics
- Smart Camera
- Smart Scanning Device
- Social networks
- Dating sites
- Tourism
- Real estate
- Manufacturing
- Biometrics
- Security
- Satellite or aerial images
- Medical
- Financial forecasting
- Robotics vision & control
- Control systems & optimization
- Autonomous vehicles
We have the following usage examples: object/face recognition; rules engines & control modules; Computation with Words & soft boundaries; classification &. search; information web; data search & organizer & data mining & marketing data analysis; search for similar-looking locations or monuments; search for similar-looking properties; defect analysis; fingerprint, iris, and face recognition; Facelemotionlexpression recognition, monitoring, tracking; recognition & information extraction, for security & map; diagnosis, using images & rules engines; and Pattern and data analysis & prediction; image ad network; smart cameras and phones; mobile and wearable devices; searchable albums and videos; marketing analytics; social network analytics; dating sites; security; tracking and monitoring; medical records and diagnosis and analysis, based on images; real estate and tourism, based on building, structures, and landmarks; maps and location services and security/intelligence, based on satellite or aerial images; big data analytics; deep image recognition and search platform; deep/detailed machine learning; object recognition (e.g., shoe, bag, clothing, watch, earring, tattoo, pants, hat, cap, jacket, tie, medal, wrist band, necklace, pin, decorative objects, fashion accessories, ring, food, appliances, equipment, tools, machines, cars, electrical devices, electronic devices, office supplies, office objects, factory objects, and the like).
Here, we also introduce Z-webs, including Z-factors and Z-nodes, for the understanding of relationships between objects, subjects, abstract ideas, concepts, or the like, including face, car, images, people, emotions, mood, text, natural language, voice, music, video, locations, formulas, facts, historical data, landmarks, personalities, ownership, family, friends, love, happiness, social behavior, voting behavior, and the like, to be used for many applications in our life, including on the search engine, analytics, Big Data processing, natural language processing, economy forecasting, face recognition, dealing with reliability and certainty, medical diagnosis, pattern recognition, object recognition, biometrics, security analysis, risk analysis, fraud detection, satellite image analysis, machine generated data analysis, machine learning, training samples, extracting data or patterns (from the video, images, text, or music, and the like), editing video or images, and the like. Z-factors include reliability factor, confidence factor, expertise factor, bias factor, truth factor, trust factor, validity factor, “trustworthiness of speaker”, “sureness of speaker”, “statement helpfulness”, “expertise of speaker”, “speaker's truthfulness”, “perception of speaker (or source of information)”, “apparent confidence of speaker”, “broadness of statement”, and the like, which is associated with each Z-node in the Z-web.
For one embodiment/example, e.g., we have “Usually, people wear short sleeve and short pants in Summer.”, as a rule number N given by an SME, e.g., human expert. The word “short” is a fuzzy parameter for both instances above. The sentence above is actually expressed as a Z-number, as described before, invented recently by Prof. Lotfi Zadeh, one of our inventors here. The collection of these rules can simplify the recognition of objects in the images, with higher accuracy and speed, e.g., as a hint, e.g., during Summer vacation, the pictures taken probably contain shirts with short sleeves, as a clue to discover or confirm or examine the objects in the pictures, e.g., to recognize or examine the existence of shirts with short sleeves, in the given pictures, taken during the Summer vacation. Having other rules, added in, makes the recognition faster and more accurate, as they can be in the web of relationships connecting concepts together, e.g., using our concept of Z-web, described before, or using semantic web. For example, the relationship between 4th of July and Summer vacation, as well as trip to Florida, plus shirt and short sleeve, in the image or photo, can all be connected through the Z-web, as nodes of the web, with Z numbers or probabilities in between on connecting branches, between each 2 parameters or concepts or nodes, as described before in this disclosure and in our prior parent applications.
In addition, there are many other embodiments in the current disclosure that deal with other important and innovative topics/subjects, e.g., related to General AI, versus Specific or Vertical or Narrow AI, machine learning, using/requiring only a small number of training samples (same as humans can do), learning one concept and use it in another context or environment (same as humans can do), addition of reasoning and cognitive layers to the learning module (same as humans can do), continuous learning and updating the learning machine continuously (same as humans can do), simultaneous learning and recognition (at the same time) (same as humans can do), and conflict and contradiction resolution (same as humans can do), with application, e.g., for image recognition, application for any pattern recognition, e.g., sound or voice, application for autonomous or driverless cars, application for security and biometrics, partial or covered or tilted or rotated face recognition, or emotion and feeling detections, application for playing games or strategic scenarios, application for fraud detection or verification/validation, e.g., for banking or cryptocurrency or tracking fund or certificates, application for medical imaging and medical diagnosis and medical procedures and drug developments and genetics, application for control systems and robotics, application for prediction, forecasting, and risk analysis, e.g., for weather forecasting, economy, oil price, interest rate, stock price, insurance premium, and social unrest indicators/parameters, and the like. (These results had been originally reported in our parent cases, as well.)
In one embodiment, we present a brief description of the basics of stratified programming (SP). SP is a computational system in which the objects of computation are in the main, nested strata of data centering on a target set, T. SP has a potential for significant applications in many fields, among them, robotics, optimal control, planning, multiobjective optimization, exploration, search, and Big Data. In spirit, SP has some similarity to dynamic programing (DP), but conceptually it is much easier to understand and much easier to implement. An interesting question which relates to neuro science is: Is the human brain employ stratification to store information? It will be natural to represent a concept such as a chair as a collection of strata with one or more strata representing a type of chair.
Underlining of our approach is a model, call it FSM. FSM is a finite state system. The importance of FSM as a model varies from use of digitalization (granulation, quantization) to almost any kind of system that can be approximated by a finite state system. The most important part is the concept of reachability of a target set in minimum number of steps. The objective of minimum number of steps serves as a basis for verification of the step of FSM state space. A concept which plays a key role in our approach is the target set reachability. Reachability involves moving (transitioning) FSM from a state w to a state in target state, T, in a minimum number of steps. To this end, the state space, W, is stratified through the use of what is called the incremental enlargement principle. Reachability is also related to the concept of accessibility.
For the current inventions, we can combine/attach/integrate/connect any and all the systems and methods (or embodiments or steps or sub-components or algorithms or techniques or examples) of our own prior applications/teachings/spec/appendices/FIGS., which we have priority claim for, as mentioned in the current spec/application, to provide very efficient and fast algorithms for image processing, learning machines, NLP, pattern recognition, classification, SVM, deep/detailed analysis/discovery, and the like, for all the applications and usages mentioned here in this disclosure, with all tools, systems, and methods provided here.
DETAILED DESCRIPTION OF THE EMBODIMENTS
This disclosure has many embodiments, systems, methods, algorithms, inventions, vertical applications, usages, topics, functions, variations, and examples. We divided them into sections for ease of reading, but they are all related and can be combined as one system, or as combination of subsystems and modules, in any combinations or just alone. We start here with the embodiment Z-number, and other inventions/embodiments will follow below after this section.
Z-Numbers:
A Z-number is an ordered pair of fuzzy numbers, (A,B). For simplicity, in one embodiment, A and B are assumed to be trapezoidal fuzzy numbers. A Z-number is associated with a real-valued uncertain variable, X, with the first component, A, playing the role of a fuzzy restriction, R(X), on the values which X can take, written as X is A, where A is a fuzzy set. What should be noted is that, strictly speaking, the concept of a restriction has greater generality than the concept of a constraint. A probability distribution is a restriction but is not a constraint (see L. A. Zadeh, Calculus of fuzzy restrictions, in: L. A. Zadeh, K. S. Fu, K. Tanaka, and M. Shimura (Eds.), Fuzzy sets and Their Applications to Cognitive and Decision Processes, Academic Press, New York, 1975, pp. 1-39). A restriction may be viewed as a generalized constraint (see L. A. Zadeh, Generalized theory of uncertainty (GTU)-principal concepts and ideas, Computational Statistics & Data Analysis 51, (2006) 15-46). In this embodiment only, the terms restriction and constraint are used interchangeably.
The restriction
R(X): X is A,
is referred to as a possibilistic restriction (constraint), with A playing the role of the possibility distribution of X. More specifically,
R(X): X is A→Poss(X=u)=μA(u)
where μA is the membership function of A, and u is a generic value of X. μA may be viewed as a constraint which is associated with R(X), meaning that μA(u) is the degree to which u satisfies the constraint.
When X is a random variable, the probability distribution of X plays the role of a probabilistic restriction on X. A probabilistic restriction is expressed as:
R(X): X isp p
where p is the probability density function of X. In this case,
R(X): X isp p→Prob(u≤X≤u+du)=p(u)du
Note. Generally, the term “restriction” applies to X is R. Occasionally, “restriction” applies to R. Context serves to disambiguate the meaning of “restriction.”
The ordered triple (X,A,B) is referred to as a Z-valuation. A Z-valuation is equivalent to an assignment statement, X is (A,B). X is an uncertain variable if A is not a singleton. In a related way, uncertain computation is a system of computation in which the objects of computation are not values of variables but restrictions on values of variables. In this embodiment/section, unless stated to the contrary, X is assumed to be a random variable. For convenience, A is referred to as a value of X, with the understanding that, strictly speaking, A is not a value of X but a restriction on the values which X can take. The second component, B, is referred to as certainty. Certainty concept is related to other concepts, such as sureness, confidence, reliability, strength of belief, probability, possibility, etc. However, there are some differences between these concepts.
In one embodiment, when X is a random variable, certainty may be equated to probability. Informally, B may be interpreted as a response to the question: How sure are you that X is A? Typically, A and B are perception-based and are described in a natural language. Example: (about 45 minutes, usually.) A collection of Z-valuations is referred to as Z-information. It should be noted that much of everyday reasoning and decision-making is based, in effect, on Z-information. For purposes of computation, when A and B are described in a natural language, the meaning of A and B is precisiated (graduated) through association with membership functions, μA and μB, respectively,
The membership function of A, μA, may be elicited by asking a succession of questions of the form: To what degree does the number, a, fit your perception of A? Example: To what degree does 50 minutes fit your perception of about 45 minutes? The same applies to B. The fuzzy set, A, may be interpreted as the possibility distribution of X. The concept of a Z-number may be generalized in various ways. In particular, X may be assumed to take values in Rn, in which case A is a Cartesian product of fuzzy numbers. Simple examples of Z-valuations are:
(anticipated budget deficit, close to 2 million dollars, very likely)
(population of Spain, about 45 million, quite sure)
(degree of Robert's honesty, very high, absolutely)
(degree of Robert's honesty, high, not sure)
(travel time by car from Berkeley to San Francisco, about 30 minutes, usually)
(price of oil in the near future, significantly over 100 dollars/barrel, very likely)
It is important to note that many propositions in a natural language are expressible as Z-valuations. Example: The proposition, p,
p: Usually, it takes Robert about an hour to get home from work,
is expressible as a Z-valuation:
(Robert's travel time from office to home, about one hour, usually)
If X is a random variable, then X is A represents a fuzzy event in R, the real line. The probability of this event, p, may be expressed as (see L. A. Zadeh, Probability measures of fuzzy events, Journal of Mathematical Analysis and Applications 23 (2), (1968) 421-427.):
where pX is the underlying (hidden) probability density of X. In effect, the Z-valuation (X,A,B) may be viewed as a restriction (generalized constraint) on X defined by:
Prob(X is A) is B.
What should be underscored is that in a Z-number, (A,B), the underlying probability distribution, pX, is not known. What is known is a restriction on pX which may be expressed as:
Note: In this embodiment only, the term “probability distribution” is not used in its strict technical sense.
In effect, a Z-number may be viewed as a summary of pX. It is important to note that in everyday decision-making, most decisions are based on summaries of information. Viewing a Z-number as a summary is consistent with this reality. In applications to decision analysis, a basic problem which arises relates to ranking of Z-numbers. Example: Is (approximately 100, likely) greater than (approximately 90, very likely)? Is this a meaningful question? We are going to address these questions below.
An immediate consequence of the relation between pX and B is the following. If Z=(A,B) then Z′=(A′,1−B), where A′ is the complement of A and Z′ plays the role of the complement of Z. 1−B is the antonym of B (see, e.g., E. Trillas, C. Moraga, S. Guadarrama, S. Cubillo and E. Castiñeira, Computing with Antonyms, In: M. Nikravesh, J. Kacprzyk and L. A. Zadeh (Eds.), Forging New Frontiers: Fuzzy Pioneers I, Studies in Fuzziness and Soft Computing Vol 217, Springer-Verlag, Berlin Heidelberg 2007, pp. 133-153.).
An important qualitative attribute of a Z-number is informativeness. Generally, but not always, a Z-number is informative if its value has high specificity, that is, is tightly constrained (see, for example, R. R. Yager, On measures of specificity, In: O. Kaynak, L. A. Zadeh, B. Turksen, I. J. Rudas (Eds.), Computational Intelligence: Soft Computing and Fuzzy-Neuro Integration with Applications, Springer-Verlag, Berlin, 1998, pp. 94-113.), and its certainty is high. Informativeness is a desideratum when a Z-number is a basis for a decision. It is important to know that if the informativeness of a Z-number is sufficient to serve as a basis for an intelligent decision.
The concept of a Z-number is after the concept of a fuzzy granule (see, for example, L. A. Zadeh, Fuzzy sets and information granularity, In: M. Gupta, R. Ragade, R. Yager (Eds.), Advances in Fuzzy Set Theory and Applications, North-Holland Publishing Co., Amsterdam, 1979, pp. 3-18. Also, see L. A. Zadeh, Possibility theory and soft data analysis, In: L. Cobb, R. M. Thrall (Eds.), Mathematical Frontiers of the Social and Policy Sciences, Westview Press, Boulder, Colo., 1981, pp. 69-129. Also, see L. A. Zadeh, Generalized theory of uncertainty (GTU)-principal concepts and ideas, Computational Statistics & Data Analysis 51, (2006) 15-46.). It should be noted that the concept of a Z-number is much more general than the concept of confidence interval in probability theory. There are some links between the concept of a Z-number, the concept of a fuzzy random number and the concept of a fuzzy random variable (see, e.g., J. J. Buckley, J. J. Leonard, Chapter 4: Random fuzzy numbers and vectors, In: Monte Carlo Methods in Fuzzy Optimization, Studies in Fuzziness and Soft Computing 222, Springer-Verlag, Heidelberg, Germany, 2008. Also, see A. Kaufman, M. M. Gupta, Introduction to Fuzzy Arithmetic: Theory and Applications, Van Nostrand. Ikeinhold Company, New York, 1985. Also, see C. V. Negoita, D. A. Ralescu, Applications of Fuzzy Sets to Systems Analysis, Wiley, New York, 1975.).
A concept which is closely related to the concept of a Z-number is the concept of a Z+-number. Basically, a Z+-number, Z+, is a combination of a fuzzy number, A, and a random number, R, written as an ordered pair ZH+=(A,R). In this pair, A plays the same role as it does in a Z-number, and R is the probability distribution of a random number. Equivalently, R may be viewed as the underlying probability distribution of X in the Z-valuation (X,A,B). Alternatively, a Z+-number may be expressed as (A,pX) or (μA,pX), where μA is the membership function of A. A Z+-valuation is expressed as (X,A,pX) or, equivalently, as (X,μA,pX), where pX is the probability distribution (density) of X. A Z+-number is associated with what is referred to as a bimodal distribution, that is, a distribution which combines the possibility and probability distributions of X. Informally, these distributions are compatible if the centroids of μA and pX are coincident, that is,
The scalar product of μA and pX, μA·pX, is the probability measure, PA, of A. More concretely,
It is this relation that links the concept of a Z-number to that of a Z+-number. More concretely,
Z(A,B)=Z+(A,μA·pXis B)
What should be underscored is that in the case of a Z-number what is known is not pX but a restriction on pX expressed as: μA·pX is B. By definition, a Z+-number carries more information than a Z-number. This is the reason why it is labeled a Z+-number. Computation with Z+-numbers is a portal to computation with Z-numbers.
The concept of a bimodal distribution is of interest in its own right. Let X be a real-valued variable taking values in U. For our purposes, it is convenient to assume that U is a finite set, U={u1, . . . , un}. We can associate with X a possibility distribution, μ, and a probability distribution, p, expressed as:
μ=μ1/u1+ . . . +μn/un
p=p1\u1+ . . . +pn\un
in which μi/ui means that μi, i=1, . . . n, is the possibility that X=ui. Similar pi\ui means that pi is the probability that X=ui.
The possibility distribution, μ, may be combined with the probability distribution, p, through what is referred to as confluence. More concretely,
μ:p=(μ1, p1)/u1+ . . . +(μn, pn)/un
As was noted earlier, the scalar product, expressed as μ·p, is the probability measure of A. In terms of the bimodal distribution, the Z+-valuation and the Z-valuation associated with X may be expressed as:
(X, A, pX)
(X, A, B), μA·pX is B,
respectively, with the understanding that B is a possibilistic restriction on μA·pX.
Both Z and Z+may be viewed as restrictions on the values which X may take, written as: X is Z and X is Z+, respectively. Viewing Z and Z+ as restrictions on X adds important concepts to representation of information and characterization of dependencies. In this connection, what should be noted is that the concept of a fuzzy if-then rule plays a pivotal role in most applications of fuzzy logic. What follows is a very brief discussion of what are referred to as Z-rules—if-then rules in which the antecedents and/or consequents involve Z-numbers or Ztnumbers.
A basic fuzzy if-then rule may be expressed as: if X is A then Y is B, where A and B are fuzzy numbers. The meaning of such a rule is defined as:
if X is A then Y is B→(X,Y) is A×B
where A×B is the Cartesian product of A and B. It. is convenient to express a generalization of the basic if-then rule to Z-numbers in terms of Z-valuations. More concretely,
if (X, AX, BX) then (Y, AY, BY)
EXAMPLES
if (anticipated budget deficit, about two million dollars, very likely) then (reduction in staff, about ten percent, very likely)
if (degree of Robert's honesty, high, not sure) then (offer a position, not, sure)
if (X, small) then (Y, large, usually.)
An important question relates to the meaning of Z-rules and Z+-rules. The meaning of a Z+-rule may be expressed as:
if (X,AX,pX) then (Y, AY, pY)→(X,Y) is (AX×AY,pXpY)
where AX×AY is the Cartesian product AX and AY
Z-rules have the important applications in decision analysis and modeling of complex systems, especially in the realm of economics (for example, stock market and specific stocks) and medicine (e.g., diagnosis and analysis).
A problem which plays a key role in many applications of fuzzy logic, especially in the realm of fuzzy control, is that of interpolation. More concretely, the problem of interpolation may be formulated as follows. Consider a collection of fuzzy if-then rules of the form:
if X is Ai then Y is Bi, i=1, . . . , n
where the Ai and Bi are fuzzy sets with specified membership functions. If X is A, where A is not one of the Ai, then what is the restriction on Y?
The problem of interpolation may be generalized in various ways. A generalization to Z-numbers may be described as follows. Consider a collection Z-rules of the form:
if X is Ai then usually (Y is Bi), i=1, . . . , n
where the Ai and Bi are fuzzy sets. Let A be a fuzzy set which is not one of the Ai. What is the restriction on Y expressed as a Z-number? An answer to this question would add a useful formalism to the analysis of complex systems and decision processes.
Representation of Z-numbers can be facilitated through the use of what is called a Z-mouse. Basically, a Z-mouse is a visual means of entry and retrieval of fuzzy data.
The cursor of a Z-mouse is a circular fuzzy mark, called an f-mark, with a trapezoidal distribution of light intensity. This distribution is interpreted as a trapezoidal membership function of a fuzzy set. The parameters of the trapezoid are controlled by the user. A fuzzy number such as “approximately 3” is represented as an f-mark on a scale, with 3 being the centroid of the f-mark (
A key idea which underlies the concept of a Z-mouse is that visual interpretation of uncertainty is much more natural than its description in natural language or as a membership function of a fuzzy set. This idea is closely related to the remarkable human capability to precisiate (graduate) perceptions, that is, to associate perceptions with degrees. As an illustration, if I am asked “What is the probability that Obama will be reelected?” I would find it easy to put an f-mark on a scale from 0 to 1. Similarly, I could put an f-mark on a scale from 0 to 1 if I were asked to indicate the degree to which I like mY job. It is of interest to note that a Z-mouse could be used as an informative means of polling, making it possible to indicate one's strength of feeling about an issue. Conventional polling techniques do not assess strength of feeling.
Using a Z-mouse, a Z-number is represented as two f-marks on two different scales (
Commutation with Z-Numbers
What is meant by computation with Z-numbers? Here is a simple example. Suppose that I intend to drive from Berkeley to San Jose via Palo Alto. The perception-based information which I have may be expressed as Z-valuations: (travel time from Berkeley to Palo Alto, about an hour, usually) and (travel time from Palo Alto to San Jose, about twenty-five minutes, usually.) How long will it take me to drive from Berkeley to San Jose? In this case, we are dealing with the sum of two Z-numbers (about an hour, usually) and (about twenty-five minutes, usually.) Another example: What is the square root of (A,B)? Computation with Z-numbers falls within the province of Computing with Words (CW or CWW). Example: What is the square root of a Z-number?
Computation with Z+-numbers is much simpler than computation with Z-numbers. Assume that * is a binary operation whose operands are Z+-numbers, Z+X=(AX,RX) and Z+Y=(AY,RY.) By definition,
Z+X*Z+Y=(AX*AY, RX*RY)
with the understanding that the meaning of * in RX*RY is not the same as the meaning of * in AX*AY. In this expression, the operands of * in AX*AY are fuzzy numbers; the operands of * in RX*RY are probability distributions.
Example: Assume that * is sum. In this case, AX+AY is defined by:
μ(A
Similarly, assuming that RX and RY are independent, the probability density function of RX*RY is the convolution, ∘, of the probability density functions of RX and RY. Denoting these probability density functions as pR
Thus,
Z+X+Z+Y=(AX+AY, pR
It should be noted that the assumption that RX and RY are independent implies worst case analysis.
More generally, to compute ZX*ZY what is needed is the extension principle of fuzzy logic (see, e.g., L. A. Zadeh, Probability measures of fuzzy events, Journal of Mathematical Analysis and Applications 23 (2), (1968) 421-427.). Basically, the extension principle is a rule for evaluating a function when what are known are not the values of arguments but restrictions on the values of arguments. In other words, the rule involves evaluation of the value of a function under less than complete information about the values of arguments.
Note. Originally, the term “extension principle” was employed to describe a rule which serves to extend the domain of definition of a function from numbers to fuzzy numbers. In this disclosure, the term “extension principle” has a more general meaning which is stated in terms of restrictions. What should be noted is that, more generally, incompleteness of information about the values of arguments applies also to incompleteness of information about functions, in particular, about functions which are described as collections of if-then rules.
There are many versions of the extension principle. A basic version was given in the article: (L. A, Zadeh, Fuzzy sets, Information and Control 8, (1965) 338-353.). In this version, the extension principle may be described as:
where A is a fuzzy set, μA is the membership function of A, μY is the memo p function of Y, and u and v are generic values of X and Y, respectively.
A discrete version of this rule is:
In a more general version, we have
For a function with two arguments, the extension principle reads:
Z=f(X,Y)
R(X): g(X) is A (constraint on u is μA(g(u)))
In application to probabilistic restrictions, the extension principle leads to results which coincide with standard results which relate to functions of probability distributions. Specifically, for discrete probability distributions, we have:
For functions with two arguments, we have:
For the case where the restrictions are Z+-numbers, the extension principle reads:
It is this version of the extension principle that is the basis for computation with Z-numbers. Now, one may want to know if f(pX,pY) is compatible with f(AX,AY).
Turning to computation with Z-numbers, assume for simplicity that *=sum. Assume that ZX=(AX,BX) and ZY=(AY,BY). Our problem is to compute the sum Z=X+Y. Assume that the associated Z-valuations are (X, AX, BX), (Y, AY, BY) and (Z, AZ, BZ).
The first step involves computation of pZ. To begin with, let us assume that pX and pY are known, and let us proceed as we did in computing the sum of Z+-numbers. Then
PZ=pX∘pY
or more concretely,
In the case of Z-numbers what we know are not pX and pY but restrictions on pX and pY
In terms of the membership functions of BX and BY, these restrictions may be expressed as:
Additional restrictions on pX and pY are:
Applying the extension principle, the membership function of pZ may be expressed as:
In this case, the combined restriction on the arguments is expressed as a conjunction of their restrictions, with A interpreted as min. In effect, application of the extension principle reduces computation of pZ to a problem in functional optimization. What is important to note is that the solution is not a value of pZ but a restriction on the values of pZ, consistent with the restrictions on pX and pY.
At this point it is helpful to pause and summarize where we stand. Proceeding as if we are dealing with Z+-numbers, we arrive at an expression for pZ as a function of pX and pY. Using this expression and applying the extension principle we can compute the restriction on pZ which is induced by the restrictions on pX and pY. The allowed values of pZ consist of those values of pz which are consistent with the given information, with the understanding that consistency is a matter of degree.
The second step involves computation of the probability of the fuzzy event, Z is AZ, given pZ. As was noted earlier, in fuzzy logic the probability measure of the fuzzy event X is A, where A is a fuzzy set and X is a random variable with probability density pX, is defined as:
Using this expression, the probability measure of AZ may be expressed as:
where
μA
It should be noted that BZ is a number when pZ is a known probability density function, Since what we know about pZ is its possibility distribution, μp
where μp
In a similar way, we can compute various functions of Z-numbers. The basic idea which underlies these computations may be summarized as follows. Suppose that our problem is that of computing f(ZX,ZY), where ZX and ZY are Z-numbers, ZX=(AX,BX) and ZY=(AY,BY), respectively, and f(ZX,ZY)=(AZ,BZ). We begin by assuming that the underlying probability distributions pX and pY are known. This assumption reduces the computation of f(ZX,ZY) to computation of f(ZX+,ZY+), which can be carried out through the use of the version of the extension principle which applies to restrictions which are Z+-numbers. At this point, we recognize that what we know are not pX and pY but restrictions on pX and pY. Applying the version of the extension principle which relates to probabilistic restrictions, we are led to f(ZX,ZY). We can compute the restriction, BZ, of the scalar product of f(AX,AY) and f(pX,pY). Since AZ=f(AX,AY), computation of BZ completes the computation of f(ZX,ZY).
It is helpful to express the summary as a version of the extension principle. More concretely, we can write:
where pX and pY are constrained by:
In terms of the membership functions of BX and BY, these restrictions may be expressed as:
Additional restrictions on pX and pY are:
Consequently, in agreement with earlier results we can write:
What is important to keep in mind is that A and B are, for the most part, perception-based and hence intrinsically imprecise. Imprecision of A and B may be exploited by making simplifying assumptions about A and B—assumptions that are aimed at reduction of complexity of computation with Z-numbers and increasing the informativeness of results of computation. Two examples of such assumptions are sketched in the following.
Briefly, a realistic simplifying assumption is that pX and pY are parametric distributions, in particular, Gaussian distributions with parameters mX, σX2 and mY, σY2, respectively. Compatibility conditions fix the values of mX and mY. Consequently, if bX and bY are numerical measures of certainty, then bX and by determine pX and pY, respectively. Thus, the assumption that we know bX and bY is equivalent to the assumption that we know pX and pY. Employing the rules governing computation of functions of Z+-numbers, we can compute BZ as a function of bX and bY, At this point, we recognize that BX and BY are restrictions on bX and bY, respectively. Employment of a general version of the extension principle leads to BZ and completes the process of computation. This may well be a very effective way of computing with Z-numbers. It should be noted that a Gaussian distribution may be viewed as a very special version of a Z-number.
Another effective way of exploiting the imprecision of A and B involves approximation of the trapezoidal membership function of A by an interval-valued membership function, Ab, where Ab is the bandwidth of A (
(AXb, BX)*(AYb, BY)=(AXb*AYb, BX×BY)
where BX×BY is the product of the fuzzy numbers BX and BY. Validity of this expression depends on how well an interval-valued membership function approximates to a trapezoidal membership function.
Clearly, the issue of reliability of information is of pivotal importance in planning, decision-making, formulation of algorithms and management of information. There are many important directions which are explored, especially in the realm of calculi of Z-rules and their application to decision analysis and modeling of complex systems.
Computation with Z-numbers may be viewed as a generalization of computation with numbers, intervals, fuzzy numbers and random numbers. More concretely, the levels of generality are: computation with numbers (ground level 1); computation with intervals (level 1); computation with fuzzy numbers (level 2); computation with random numbers (level 2); and computation with Z-numbers (level 3), The higher the level of generality, the greater is the capability to construct realistic models of real-world systems, especially in the realms of economics, decision analysis, risk assessment, planning, analysis of causality and biomedicine.
It should be noted that many numbers, especially in fields such as economics and decision analysis are in reality Z-numbers, but they are not currently treated as such because it is much simpler to compute with numbers than with Z-numbers. Basically, the concept of a Z-number is a step toward formalization of the remarkable human capability to make rational decisions in an environment of imprecision and uncertainty.
Analysis Methods using Probability Distributions with Z-Number:
We discussed the probability measure of a fuzzy set A in RX based on a hidden probability distribution pX, is determined as
In evaluation of Z number, this probability measure is restricted by a fuzzy set B, with the restriction determined by
The restriction is then implied on the probability distribution. In an example shown in
In one embodiment, as depicted for example in
In one embodiment, a particular class/template/type of probability distribution is selected to extend the restriction on pX onto restriction on pX's parameters. For example, in one embodiment, a normal or Gaussian distribution is taken for pX (as shown in
For pX as N(mx, σx), the above probability measure of A, is reduced to expression with erf and exp terms with mx, σx and r. In one embodiment, the probability measures are pre-determined/calculated/tabulated for various values of mx, σx and r. Note that any demoralization on X does not affect the probability measure, while a denormalization in μA(x) (i.e., maximum membership value) scales the probability measure.
In one embodiment, (pX·μX) (here denoted as ν) is determined and/or stored in a model database, for various pX. For example, ν is depicted versus σx in
In one embodiment, based on μBx(ν), for various ν's (e.g., νmin, νmid, and/or νmax), close pX's or (mx, σx)'s candidate are determined, e.g., by tracking/determining the contour lines, via (mesh) interpolation using test (or random) pX's or (mx, σx) (e.g., by using a root finding method such as Secant method). In one embodiment, these subsets of pX's or (mx, σx) reduce the computation resources needed to apply the restriction on other variables or probability distributions.
For example, in a setting where Y=F(X), Z-valuation (X, Ax, By) may be extended to (Y, Ay, By) through restrictions on pX. In one embodiment, where Ay is determined via extension principle using F(X) and Ax, By is determined by finding the restrictions on probability measure of Ay. In one embodiment, F(X) is monotonic, i.e., X=F−1(Y) is unique.
where δxy is (+1) if F(X) is (monotonically) increasing and it is (−1) if F(X) is decreasing.
The extension principle also provides that, μAx(x) is μAy(y), where y=F(x). Therefore, the probability measure of Ay, denoted as ω (=pY·μAy), becomes the same as ν, for the same px or (mx, σx), as shown below:
Therefore, μBy(ω) becomes identical to μBx(ν) (for any candidate pX), when F(X) is monotonic and Ay is determined via extension principle from Ax and F(X). This result does not hold when F(X) is not monotonic, but it may be used as first order approximation, in one embodiment. For example, for non-monotonic F(X), still assuming Ay is determined via extension principle from Ax and F(X):
Suppose in Y domain, there are N piecewise monotonic regions of F(X). Therefore, there are up to N number of x's as solutions to F−1(y), denoted by a set {x1, . . . , xi, . . . , xN}. An event occurring in Y domain, may occur at any of {xi}, therefore
where δxy,i indicates, as before, whether ith monotonic region of F(X) is increasing or decreasing.
In an embodiment, ω is determined by:
where x′ ∈{xi }. Therefore,
In one embodiment, where, e.g., due to relative symmetry in F(X) and μAx(x), μAx(x) is the same for ∀x′ ∈{xi}, then ω=ν, because
for any xi.
Likewise, in one embodiment, where μAx(x) is zero or negligible in a region (e.g., for N=2), then ω=ν, as the contribution to ω comes from the dominant monotonic region of F(X).
In one embodiment, deviation of ω from ν is estimated/determined by determining difference between
and various μA
In one embodiment, where μAy(y) is provided via a proposition (instead of being determined via extension principle through F(X) and Ax), μA·y(y) is determined (via extension principle) and compared to μAy(y). If there is a match, then ω is estimated using ν, e.g., as described above.
In one embodiment, as for example depicted in
One embodiment, for example, uses the concepts above for prediction of stock market, parameters related to economy, or other applications. Consider the following example:
Example 1
We are given this information (for anticipation and prediction): There probability that the price of oil next month is significantly over 100 dollars/barrel is not small.
Assume that the ticket price for an airline from Washington DC to New York is in the form of (Y=F(X)=a1·X+a2), where X is the next month's estimated price of oil (in dollars/barrel) and Y is the ticket price (in dollars). For this example, further assume that a1=1.5 and a2=150, i.e., Y=1.5 X+150. Then, we have the following questions:
q1: What is the Price of the Ticket from Washington DC to New York?
X represents (the price of oil the next month), Ax is (significantly over 100 dollars/barrel) and BX is (not small). Then, (X, Ax, Bx) is a Z-valuation restricting the probability of(X) the price of oil the next month. In this example, as depicted in
In one embodiment, the probability measure of Ax, (i.e., ν), is determined by dot product of pX and μAx. In one embodiment, pX is evaluated at x values in X domain (e.g., against a set of points between xmin and xmax). Similarly, μAx is determined at the data set {x1} in X domain (or at significant, e.g., corner points of μAx). In one embodiment, the dot product is determined by evaluating
νp
In one embodiment, ν is determined via piecewise evaluation (e.g., using exp and erf functions when pX is Gaussian). In one embodiment, ν is determined for various candidates for pX. For example, taking pX, as N(mx, σx) as described above, ν is determined for various (mx, σx) combination, as depicted in
Given restriction not small, Bx, in one embodiment, the test score for each candidate pX is evaluated, by evaluating the truth value of its corresponding probability measure of Ax, ν, in μBx(ν). In one embodiment, the assignment of test score is used for pX candidates corresponding to a particular set of ν values (e.g., those used to define μBx(ν) such as the ramp location or trapezoid corners). In such an embodiment, bins are associated with such particular ν's to determine pX candidates with corresponding ν values within a bin. Those candidates, are for example, identified by those (mx, σx) at or near particular contour lines of interest (e.g., marked as ν1, ν2, and ν3 at ν values of 0, 0.25 and 0.5, on
In one embodiment, the probability measure of Ay, (i.e., ω), is determined by dot product of pY and μAy. In one embodiment, pY is determined via application of extension principal. In one embodiment, pX's for points in {xi} in X domain are attributed to their corresponding points {yi} in Y domain. Such an embodiment accommodates having multiple yi's have the same value (or belong to the same bin in Y domain). Alternatively, or additionally, in one embodiment, bins are setup in Y domain to determine pY for each bin by summing over corresponding pi's (from X domain) where F(xi) is within the Y-bin. In such an embodiment, ω, for example, is determined by taking pY and μAy dot product in Y domain over Y bins. However, in one embodiment, pY and μAy dot product is essentially determined in X domain, for example by:
ωp
In one embodiment, ω is determined via piecewise evaluation. In one embodiment, ω is determined for various candidates for pX. For example, taking pX, as N(mx, σx) as described above, ω is determined for various (mx, σx) combination, as depicted in
In one embodiment, to obtain the relationship between ω and restriction test scores from Bx, to determine By, bins are setup in ω domain (e.g., between ωmin and ωmax, or in [0, 1] range). In one embodiment, the size/number of bin(s) in ω is adjustable or adaptive to accommodate regions in ω domain where (mx, σx) mapping is scarce, sparse or absent. In one embodiment, for each (mx, σx), the calculated ω (mx, σx), is mapped to a bin in ω domain. In such an embodiment, each (mx, σx) becomes associated to a ω bin (e.g., identified by an ID or index). Multiple (mx, σx) may map to the same ω bin. In one embodiment, through this association with the same ω bin, the maximum μBx(ν(mx, σx)) for (mx, σx)'s associated with the same ω bin is determined. For example,
Therefore, in this example, assuming that μAy(y) (ramping up from 300 to 345) indicates somewhat higher than 300, and that μBy(ω) maps to more than medium (i.e., not small) (in this context), then the answer to q1 becomes: The probability of the price of the ticket being somewhat higher than 300 is more than medium.
q2: What is the Probability that the Price of the Ticket (from Washington DC to New York) is not Low?
In this question, Y still presents the price of the ticket; however, Ay is already specified by q2 as not low in this context. Parsing the question, Prob(Y is Ay) or By in Z-valuation of (Y, Ay, By) is the output. In one embodiment, the knowledge database is searched to precisiate the meaning of not low in the context of Y. In one embodiment, in parsing q2, not is recognized as the modifier of a fuzzy set low in context of Y. In one embodiment, the knowledgebase is used to determined, for example low is a step down fuzzy set with its ramp located between 250 and 300. In one embodiment, the modifiers are used to convert the membership functions per truth system(s) used by the module. For example,
In one embodiment, the association of (xi, yi) is used to attribute pX values to (xi, yi). Comparing with q1. In one embodiment, ν and μAx are reused or determined similarly. For example,
In one embodiment, the resulting μBy(ω) is provided to other modules that take membership function as input (e.g., a fuzzy rule engine) or store in a knowledge data store. In one embodiment, the resulting μBy(ω) (e.g., in
In one embodiment, μBy(ω) (e.g., in
q3: What is the Probability that the Price of the Ticket (from Washington DC to New York) is High?
As in q2, q3 presents Ay as high. In one embodiment, within the context, μAy is given, for example, as ramp located at 350 (with a width of 50), as depicted in
Fuzzy Control with Z-Number:
As mentioned previously, an extension of a fuzzy control system that uses fuzzy rules can employ Z-numbers a either or both antecedent and consequent portion of IF THEN fuzzy rule. Regularly, in executing a fuzzy rule, such as (IF X is A THEN Y is B), the value of variable X used in antecedent, is determined (e.g., from an input or from defuzzification result of other relevant rules) to be x0. In one embodiment, the truth value of the antecedent is evaluated given the knowledge base (e.g., X=x0) as the truth value of how (X is A) is satisfied, i.e., μA(x0). The truth value of the antecedent (assuming more than a threshold to trigger the consequent) is then applied to the truth value of the consequent, e.g., by clipping or scaling the membership function of B by μA(x0). Firing of fuzzy rules involving the same variable at the consequent yields a superimposed membership function for Y. Then, a crisp value for Y is determined by defuzzification of Y's resulting membership function, e.g., via taking a center of mass or based on maximum membership value (e.g., in Mamdani's inference method), or a defuzzied value for Y is determined by a weighted average of the centroids from consequents of the fuzzy rules based on their corresponding truth values of their antecedents (e.g., in Sugeno fuzzy inference method).
In one embodiment, where the antecedent involves a Z-number, e.g., as in the following fuzzy rule:
IF (X is Z) THEN (Y is C), where Z=(AX, BX) and X is a random variable,
the truth value of the antecedent (X is Z) is determined by how well its imposed restriction is satisfied based on the knowledge base. For example, if the probability or statistical distribution of X is pX, the antecedent is imposing a restriction on this probability distribution as illustrated earlier as:
where u is a real value parameter in X domain. In one embodiment, the probability distribution of X, pX, is used to evaluate the truth value of the antecedent, by evaluating how well the restriction on the probability distribution is met. In one embodiment, an approximation for pX is used to determine the antecedent's truth value. Denoting pXi as an estimate or an input probability distribution for X, the antecedent truth value is determined as:
An embodiment, e.g., in a fuzzy control system or module, uses multiple values of u to estimate pX. In one embodiment, the values of u are discrete or made to be discrete through bins representing ranges of u, in order to count or track the bin population representing the probability distribution of X. For example, at bini, pX is estimated as:
where Δui and Counti are the width and population of ith bin. This way, a running count of population of bins is tracked as more sample data is received.
In one embodiment, Z-number appears as the consequent of a fuzzy rule, e.g.,
IF (Y is C) THEN (X is Z), where Z=(AX, BX) and X is a random variable.
As other fuzzy rules, when the rule is executed, the truth value of the antecedent (i.e., μC(y0), where y0 is a value for Y, that is input to the rule) is applied to the restriction imposed by the consequent. The restriction imposed by the consequent is, e.g., on the probability distribution of X, which is the variable used in the consequent. Given the antecedent's truth value of Tant (between 0 and 1), in one embodiment, the contribution of the rule on the restriction of pX is represented by
μB
In one embodiment, Z-number appears in an antecedent of a fuzzy rule, but instead of the quantity restricted (e.g., pX), other indirect knowledge base information may be available. For example, in the following fuzzy rule:
IF (X is Z) THEN (Y is C), where Z=(AX, BX) and X is a random variable,
suppose from input or other rules, it is given that (X is D), where D is a fuzzy set in X domain. In one approach, the hidden candidates of pX (denoted by index i) are given test scores based on the knowledge base, and such test scores are used to evaluate the truth value of the antecedent. For example, the truth value of the antecedent is determined by:
In one embodiment, various model(s) of probability distribution is employed (based on default or other knowledge base) to parameterize ∀i . For example, a model of normal distribution may be assumed for pX candidates, and the corresponding parameters will be the peak location and width of the distribution. Depending on the context, other distributions (e.g., Poisson distribution) are used. For example, in “Bus usually arrives about every 10 minutes”, where X is bus arrival time, AX is about 10 minutes, and BX is usually, a model of probability distribution for bus arrival time may be taken as a Poisson distribution with parameter τ:
Then, the antecedent truth value is determined by
In one embodiment, the truth value of the antecedent in a fuzzy rule with Z-number, e.g.,
IF (X is Z) THEN (Y is C), where Z=(AX, Bx) and X is a random variable,
is determined by imposing the assumption that the probability distribution pX is compatible with the knowledge base possibility restriction (e.g., (X is D)). Then, a candidate for pX may be constructed per μD. For example, by taking a normalized shape of possibility distribution:
In one embodiment, the compatibility assumption is used with a model of distribution (e.g., based on default or knowledge base). For example, assuming a model of normal distribution is selected, the candidate probability distribution is determined as follows:
where Dwidth and Dcent are the width and centroid location of (e.g., a trapezoid) fuzzy set D, and r is a constant (e.g., 1/√{square root over (12)}≈0.3) or an adjustable parameter.
In one embodiment, the truth value of the antecedent in a fuzzy rule with Z-number, e.g.,
(X is Z) THEN (Y is C), where Z=(AX, BX) and X is a random variable,
is determined by simplifying the ∀i examination in
by taking a candidate for pX based on a model of probability distribution which would be compatible with fuzzy set B. Then, the antecedent truth value is determined based on such compatible probability distribution po, as Tant=tso∧ts′o.
In one embodiment, such optimized probability distribution is determined based on the knowledge base (e.g., X is D). For example, when the model distribution is a normal distribution, in one embodiment, the center position (parameter) of the distribution is set at the centroid position of the fuzzy set D, while the variance of the probability distribution is set based on the width of fuzzy set D.
In one embodiment, an input proposition in form of Z-valuation, e.g., (X, AX, BY) or (X is Z) where Z=(AX, BY) and X is a random variable, is used to evaluate an antecedent of a fuzzy rule, e.g.,
IF (X is C) THEN (Y is D), where C and D are fuzzy sets in X and Y domains, respectively.
In one embodiment, candidates of pX (denoted by index i) are given test scores based on the knowledge base, and such test scores are used to evaluate the truth value of the antecedent. For example, in one embodiment, the truth value of the antecedent is determined by:
Example 2
In one embodiment, a fuzzy rules database includes these two rules involving Z-valuation (e.g., for a rule-based analysis/engine). Rule 1: if the price of oil is significantly over 100 dollars/barrel, the stock of an oil company would most likely increase by more than about 10 percent. Rule 2: If the sales volume is high, the stock of an oil company would probably increase a lot. There is also this input information: The price of oil is at 120 dollars/barrel; the sales volume is at $20B; and the executive incentive bonus is a function of the company's stock price. The query or output sought is:
q4: What is the Likelihood of High Executive Incentive Bonuses?
In one embodiment, the rules engine/module evaluates the truth value of the rules' antecedents, e.g., after the precisiation of meaning for various fuzzy terms. For example, the truth value of Rule 1's antecedent, the price of oil is significantly over 100 dollars/barrel is evaluated by taking the membership function evaluation of 120 (per information input) in fuzzy set significantly over 100 dollars/barrel (see, e.g.,
where j is an index for triggered fuzzy rule (in this example, from 1 to 2). As an example, in a control system employing correlation product and additive inference, the restriction on pX(mx, σx) is determined as follows, e.g., for various (mx, σx):
In one embodiment, μp
Y=G(SP)=G(CSP+X)=F(X)
In one embodiment, as in the previous examples, ω, probability measure of Ay is determined for various pX (i.e., (mx, σx)) candidates. In one embodiment, maximum μpx(mx, σx) for ω (or ω bin) is determined, and applied as membership function of μBy(ω). In another word, in this example, the output of rules engine provides the restriction on pX (or its parameters) similar to previous examples, and this output is used to determine restriction on a probability measure in Y.
Example 3
In one embodiment, e.g., in a car engine diagnosis, the following natural language rule “Usually, when engine makes rattling slapping sound, and it gets significantly louder or faster when revving the engine, the timing chain is loose.” is converted to a protoform, such as:
In one embodiment, a user, e.g., an expert, specifies the membership of a particular engine sound via a user interface, e.g., the user specifies that the truth value of the engine sound being Rattling-Slapping is 70%. In one embodiment, the user specifies such truth value as a fuzzy set, e.g., high, medium, very high. In one embodiment, a Z-mouse is used to specify the fuzzy values (i.e., membership function) of various attribute(s) of the sound (e.g., loudness, rhythm, pitch/squeakiness). The Z-mouse is for example provided through a user interface on a computing device or other controls such as sliding/knob type controls, to control the position and size of an f-mark.
In one embodiment, the engine sound is received by a sound recognition module, e.g., via a microphone input. In one embodiment, the loudness (e.g., average or peak or tonal) of the engine sound is determined, e.g., by a sound meter (analog or digital) or module. In one embodiment, the rhythm is determined via the frequency of the loudness, or using the frequency spectrum of the received sound (e.g., the separation of the peaks in the frequency domain corresponds to the period of (impulse) train making up the rhythm of the engine sound). In one embodiment, the values of these parameters are made fuzzy via evaluating the corresponding membership functions (of e.g., engine sound level) for evaluating the truth value of the predicate in fuzzy rule. In one embodiment, the fuzzy rule is rewritten to use more precision, e.g., if readily available. For example, in one embodiment, level(sound(revved.engine)) and level(sound(revved.engine)) take on measured values.
In one embodiment, as for example depicted in
where i is an index identifying the ith signature parameter Pi. ui is a generic truth value parameter in [0, 1]. tsi is the test score contribution from comparison of A and B against Pi. μA,Pi and μB,Pi are fuzzy values of the A and B with respect to signature parameter Pi. For example, A represents RattlingSlapping; B represents the engine sound; ts represents the truth value of the engine sound being RattlingSlapping; and tsi represents a possibility test score match of A and B with respect to the signature (fuzzy) parameter Pi, for example determined, by comparison of A's and B's truth degree in Pi. In one embodiment, the comparison with respect to Pi is determined by:
For example, as depicted in
In one embodiment, where not all signature parameters are used, relevant, or available for A, then a subset of those signature parameters that are used, relevant, or available for A is used to determine ts, e.g., by limiting taking minimum or averaging operations based on those signature parameters. For example,
In such an embodiment, the relevant signature parameters for A are identified, for example, via a query in the model or knowledge database.
In one embodiment, for example, when minimum of tsi's are used to determine ts, the irrelevancy of a signature parameter with respect to A may be expressed as a truth membership function of 1 for all possibilities. For example, as depicted in
In one embodiment, μA,Pi is determined through empirical methods, user settings, or training sets. For example, in one embodiment, N training set engine sounds (denoted as Tk with k from 1 to N) are used to determine μA,Pi. In one embodiment, the truth values for the training element Tk with respect to signature parameters are determined (e.g., as a crisp number, range, or a fuzzy set). For example, as depicted in
In one embodiment, user/expert assigns verity membership values for Tk in A. In one embodiment, a module is used to determine correlation between the various type sounds and the corresponding engine diagnosis (by for example experts). In one embodiment, the correlation is made between the signature parameters of the sound and the diagnosis (e.g., in for of fuzzy graphs or fuzzy rules). In one embodiment, a typical and highly frequent type of sound may be identified as the signature parameter (e.g., RattlingSlapping may be taken as a signature parameter itself). Therefore, in one embodiment, the creation of new signature parameters may be governed by fuzzy rules (e.g., involving configurable fuzzy concepts as “typical” for similarity and “frequent”). In one embodiment, the reliability and consistency of the rules are enhanced by allowing the training or feedback adjust μA,Pi.
In one embodiment, such diagnosis is used an autonomous system, e.g., in self-healing or self-repair, or through other systems/subsystems/components.
In one embodiment provides music recognition via similar analysis of its signature parameters and comparison against those from a music library/database. In one embodiment, the categories of music (e.g., classic, rock, and the like) may be used as fuzzy concept A in this example.
q5: What is the Probability of Loose Timing Chain, when the Engine Sound is a Loud “Tick, Tick, Tack, Tack” and it Gets Worse when Revving the Engine?
In one embodiment, as shown by q5, the specification of an input to the system is not in form of the actual sound engine (e.g., wave form or digitized audio), but a fuzzy description of the sound. A conversion process evaluates the fuzzy description to find or construct a sound/attributes (e.g., in the data store) which may be further processed by the rules. For example, in one embodiment, within the context, the module interprets fuzzy descriptions “Tick” and “Tack” as a tonal variation of abrupt sound. In one embodiment, the sequence of such descriptions is interpreted as the pattern of such sounds. With these attributes, in one embodiment, signature parameters are determined, and as described above, the test score related to whether “Tick, Tick, Tack, Tack” is RattlingSlapping is determined. The evaluation of the fuzzy rule predicate provides the test score for the limiting truth score for the consequent, which is a restriction on the probability of loose timing chain.
In one embodiment, e.g., in music recognition, similar fuzzy description of music is used to determine/search/find the candidates from the music library (or metadata) with best match(es) and/or rankings. When such a description accompanies other proposition(s), e.g., a user input that “the music is classical”, it would place further restrictions to narrow down the candidates, e.g., by automatic combinations of the fuzzy restrictions, as mentioned in this disclosure or via evaluation of fuzzy rules in a rules engine.
Example 4
In this example, suppose these input propositions to system: p1: the weather is seldom cold or mild. p2: Statistically, the number of people showing up for an outdoor swimming pool event is given by function having a peak of 100 at 90° F., where X is the weather temperature:
q6: How Many People Will Show Up at the Swimming Event?
In one embodiment, the precisiation of input proposition is in Z-valuation (X, Ax, Bx), where Ay is cold or mild and By is seldom. For example, as depicted in
In one embodiment, the parsing of q6 results in an answer in form of Z-valuation, (Y, Ay, By) form, where Y is the number of people showing up for an outdoor swimming pool event. In one embodiment, as described in this disclosure, a candidate μAy is determined using F(X) and μAx via extension principle. For example, as depicted in
q7: What are the Odds that the Weather is Hot?
In one embodiment, the answer is in a Z-valuation Y, Ay, By) form, where Y is temperature (same as X, i.e., Y=F(X)=X), q6 provides Ay as hot, as for example depicted in
q8; What are the Odds that More than About 50 People Show Up?
In one embodiment, the answer is in a Z-valuation Y, Ay, By) form, where Y is again the number of people showing up for an outdoor swimming pool event, and A, is more than about 50. In one embodiment, μAy is determined from q8, e.g., by using the model database and fuzzy logic rules for modifiers within the context and domain of Y, for example, as depicted in
q9: What are the Odds that More than About 65 People Show Up?
In one embodiment, similarly to above, μAy is determined to be a step up membership function with a ramp from 55 to 65, as depicted in
q10: What are the Odds that about 30 People Show Up?
In one embodiment, similarly to above, μAy is determined to be a triangular membership function with a base from ramp from 20 to 40, as depicted in
Confidence Approach on Membership Function:
As mentioned earlier, in the Z-valuation (X, A, B), a restriction on X (e.g., assuming X is a random variable), in one embodiment, is imposed via a restriction on its probability distribution pX, to the degree that the probability measure of A, defined as
(u)pX (u)du, satisfies the restriction that (Prob(X is A) is B). In such a case, pX is the underlying (hidden) probability dens of X. In one embodiment, this approach takes a view that such Z-valuation is based on an objective evaluation against the probability distribution px. In the following, we consider the view that B does not necessarily impose a restriction on pX, but on A itself. For example, B can be viewed as the confidence level on the speaker of the proposition. For example, while there may be absolutely no confidence on the propositions generated out of a random fortune teller machine, some of the propositions themselves may in fact be true or highly probable. In such a case, the confidence level imposed on the propositions have more to do with confidence in the source of the propositions rather than restriction on the probability distributions related to the random variables associated with the content of the propositions. In another example, take the proposition “Fred's height is medium height, but I am not too sure (because I don't recall too well).” In one embodiment, we take such proposition (as a matter of degree) to allow Fred's height to be medium-high or medium low. In essence, the restriction from B, in this approach, is imposed not necessarily on pX, but on imprecision of A itself. In one embodiment, this approach provides a method to deal with seemingly conflicting propositions, for example by discounting the confidence levels on such propositions (or, for example, on the speakers of those propositions), as opposed to imposing conflicting restrictions on pX.
As shown in
In one embodiment, as depicted for example in
In one embodiment, the possibility map of membership function of x associated with A* may be determined by superimposing all possible membership functions of x with their corresponding membership degree (or test score) in A* on (x, η) plane, for example, by taking the supreme test score membership degree in A*) of such potential membership functions for each point in (x, η) plane.
As depicted in
In one embodiment, as for example depicted in 24, the membership function η, μA.(X0, η), for X value of X0, revolves around no, which is the value of membership function of X in A at X0 (i.e., η0=μA(X0)). In one embodiment, the shape of μA*(X0, η) depends on B and X0. In one embodiment, the shape of μA*(X0, η) depends on B and η0. In such an embodiment, for two values of X, e.g., X1 and X4 (for example, as depicted in
In one embodiment, as depicted for example in
In one embodiment, as depicted in
In one embodiment, as depicted in
In one embodiment, the consistency of “X is C” against (A, B) is evaluated based on the degree in which C satisfies or is consistent with A* at various values of X. In one embodiment, the lowest value of such degree is taken as the degree in which C satisfies (A, B):
μA
In one embodiment, with μA*(X0, η) expressed as μη0, B(η), where η0 is μA(X0),
μA
In one embodiment, the consistency of “X is C” against (A, B) is evaluated based on the degree in which C overall satisfies or is consistent with A* by taking an average or a weighted average of the consistency of C with A* over all X:
where N is a normalization factor and W(x) is a weight factor. In one embodiment, W(x) is one for all X. In one embodiment, W(x) is a function of !.1.A(X). In one embodiment, W(x) is high for low or high membership values of μA(X), and it is low for intermediate values of μA(X). The normalization factor is then:
The above relationships may be expressed in sigma form instead of integral if X is a discrete type variable.
In one embodiment, as depicted in
ts(A″Z)=μA
In one embodiment, multiple pairs of membership functions in X and Y may map to the same membership function in Z. For example as depicted in
subject to the possibility distribution of X and Y being A′x and A′y, respectively, and Z=f(X,Y), map to a possibility distribution of Z as A′z.
Therefore, in an embodiment, possible membership functions of X and Y, belonging to fuzzy sets Ax* and Ay*, are used to determine the corresponding membership functions of Z, with their degrees of membership in Ax* determined via extension principle (from the degrees of membership of the possible membership functions of X and Y in fuzzy sets Ax* and Ay*, respectively).
In one embodiment, the set of resulting membership functions of Z (e.g., A′z) with their corresponding test score (e.g., ts(A′z)) are used to setup a fuzzy map (Az*) describing the membership function of Z:
In another words, in one embodiment, for all possible A′z, passing through point (z, η), the maximum corresponding test score is used to assign the fuzzy membership value of Az* for that point. In one embodiment, A′x, and A′y candidates are iteratively used to determine the corresponding A′z. Then, a corresponding test score for A′, is determined based on membership values of A′x and A′y candidates in Ax* and Ay*, respectively. To drive the mapping Az*, in one embodiment, (z, η) plane is granulized into segments (e.g., pixels or granules). In one embodiment, as depicted in
In one embodiment, instead of taking the approach from candidate membership functions from X and Y domain to arrive at resulting membership function at Z domain, candidates are taken from X and Y domain themselves to arrive at Z domain directly. Where the membership functions in X and Y are crisp (e.g., Ax and Ay), the resulting membership function in Z has the following form:
When the membership functions in X and Y are themselves fuzzy (e.g., Ax* and Ay*), the resulting map in Z domain, in one embodiment, is expressed as:
Or alternatively expressed as:
In one embodiment, fuzzy maps in X and Y domains are scanned, and μAz*(z, η) is determined by granularizing (z, η) to (zg, ηg) as described above and illustrated in
In one embodiment, the fuzzy map is derived based on candidate fuzzy sets in X and Y (each having same color/grayscale along its trace, e.g., based on color/grayscale contour of fuzzy maps Ax* or Ay*) and/or using alpha-cut approach in membership functions of candidate fuzzy sets from Ax* and/or Ay* (e.g., explained in this disclosure) to derive candidate fuzzy sets and their associated color/grayscale representing Az* in Z.
In one embodiment, a derived fuzzy map, such as Az* mentioned above, is used to test consistency against a candidate A. Above, a method to derive the test score for such consistency was provided. In one embodiment, a fuzzy map based on such a candidate Az is used to determine the consistency of a pair (Az, Bz) against a derived map Az*. In one embodiment, the confidence level Bz is determined so that (Az, Bz) is a representative approximation of derived map Az*. As depicted in
In one embodiment, the fuzzy maps are compared for consistency over (x and η), e.g., by comparing color/gray scale at corresponding points/granular. In one embodiment, weight is assigned to such comparison where the color/gray scale difference or the possibility of such membership value in each map is large. In one embodiment, the test score comparison between fuzzy maps is determined by point-wise coverage (e.g., with weight). In one embodiment, a threshold or a fuzzy rule is used to get point-wise coverage degree through summation or integration over map or portion of the map (e.g., where A* is above a threshold).
In one embodiment, as for example depicted in
In one embodiment, as depicted in
In one embodiment, as depicted in
In one embodiment, Cα,L and Sα L are further dependent on xi or μA(xi).
In one embodiment, a fuzzy map A* is constructed by both lateral and vertical fuzziness of A by an amount determined by B. In one embodiment, for example as depicted in
In one embodiment, the fuzzy region μxi,ηi,α(x, η) is selected to decouple (x, η) into vertical and horizontal fuzzy components, e.g.:
μx
In one embodiment, the above test is limited to set of signature points (e.g., defining the corners of μAx, or certain pre-defined values of η). In such an embodiment, color/grey scale contours (e.g., convex) are determined to envelope neighboring (x′, η′) points having the same assigned μA*(x′, η40 ) value. The envelopes are then assigned the common color/grey scale value of μA*(x′, η′). In one embodiment, these envelops of contours define μA*(x, η).
Example 5
In one embodiment, a fuzzy rules engine employs a fuzzy rule with A* at its antecedent. E.g.,:
IF (X is A*) THEN (Y is C), where A*=(i AX, BY).
In one embodiment, an input proposition, e.g., X is D, is used to evaluate the truth value (Tant) of the rule's antecedent. In one embodiment, Tant is determined based on the coverage of A* against D, such as a test score. In one embodiment, Tant is determined from (μA*∧μD), as illustrated in
In one embodiment, a centroid location of μmax(ηmin) (depicted as ηc in
Generalization of Some of the Concepts: (a) Apparent Confidence of a Speaker:
For example, let's start from the following statement: “Event A is very rare”. Let's consider the following situation: Person B (a source of information, or the speaker, or the writer) says: “Event A is very rare, and I am sure about it.”. In this example, the word “rare” signifies the statistical frequency of the event A happening. “Being sure about the statement above” indicates the “apparent” confidence of the speaker (person In this case, the degree of the “apparent confidence of the speaker” is high. Please note that this is just the “apparent” confidence of the speaker, and it may not be the “real” confidence of the speaker, due to the parameters mentioned below, such as speaker's truthfulness (which can make the apparent confidence different from the real confidence of the speaker).
In one model, the degree of the apparent confidence of the speaker is set between 0 and 1, as a normalized axis (or scale), for example, corresponding to zero (minimum) apparent confidence of the speaker level and maximum apparent confidence of the speaker level, respectively.
Please note that sometimes, the speaker only says “Event A is very rare.”, and he does not mention “and I think it is true.” in his statement. However, a listener may conclude that the speaker meant to say that “Event A is very rare, and I think it is true.”, which may be understood from the context of the statement by the speaker.
(b) Speaker's Truthfulness:
In one embodiment, person B (the speaker) might have a bias or bad faith, or may be a liar (e.g., for the statement “Event A is very rare.”). For example, he may lie very often, or he may lie often only on a specific subject or in a specific context. Or, we may have a history of lies coming from person B (as a source of information). In all of these cases, the person B “intentionally” twists his own belief, when he expresses his statement verbally or in writing. Of course, if his own belief is false (in the first place), the end result (his twisted statement) may become valid or partially valid, anyway. Thus, for any speaker who is biased, has a bad faith, or is a liar, the degree of the “speaker's truthfulness” is low. The degree of the “speaker's truthfulness” is usually hidden or unknown to the listener or reader.
In one model, the degree of the truthfulness of the speaker is set between 0 and 1, as a normalized axis (or scale), for example, corresponding to zero (minimum) and maximum truthfulness of the speaker levels, respectively. For example, 0 and 1 correspond to the always-“liar” and always-“not-liar” speakers, respectively.
Please note that the “truthfulness of a statement” is different from the “truthfulness of a speaker”.
(c) Expertise of the Speaker:
Another factor is the degree of expertise or knowledge of a person about a subject (or how well a person can analyze the data received on a given subject, or how well a person can express the ideas and conclusions to others using the right language and phrases). For example, if the event A is about astronomy and the speaker has low or no knowledge about astronomy, then the “degree of expertise of the speaker” (or source of information) is low. In one model, the degree of the expertise of the speaker is set between 0 and 1, or 0 to 100 percent, as a normalized axis (or scale), for example, corresponding to zero (minimum) and maximum expertise levels, respectively.
(d) Perception of the Speaker:
Another factor is the degree of “perception of the speaker” about an event or subject. For example, a person with a weak eye sight (and without eyeglasses) cannot be a good witness for a visual observation of an event from a far distance, for example as a witness in a court. In one model, the degree of the perception of the speaker is set between 0 and 1, as a normalized axis scale), for example, corresponding to zero (minimum) and maximum levels, respectively.
(e) Trustworthiness of a Speaker:
Now, here is a new parameter, the “trustworthiness of a speaker”, which depends on at least the 4 factors mentioned above:
-
- 1—the degree of the “apparent confidence of the speaker”
- 2—the degree of the “speaker's truthfulness”
- 3—the degree of “expertise of the speaker”
- 4—the degree of “perception of the speaker”
For example, as shown in
-
- 1—the degree of the “apparent confidence of the speaker” is high &
- 2—the degree of the “speaker's truthfulness” is high &
- 3—the degree of “expertise of the speaker” is high &
- 4—the degree of “perception of the speaker” is high
In one model, the degree of the “trustworthiness” of a speaker is set between 0 and 1, as a normalized axis (or scale), for example, corresponding to zero (or minimum) and maximum trustworthiness levels, respectively,
Please note that, in some situations, the “apparent confidence of the speaker” may become dependent or intertwined on the statement itself or one of the other parameters mentioned above, e.g., the “perception of the speaker”.
(f) Sureness of a Speaker:
Similarly, here is another parameter, the “sureness” of a speaker, which depends on at least the 4 factors mentioned above:
-
- 1—the degree of the “apparent confidence of the speaker”
- 2—the degree of the “speaker's truthfulness”
- 3—the degree of “expertise of the speaker”
- 4—the degree of “perception of the speaker”
For example, as shown in
-
- 1—the degree of the “apparent confidence of the speaker” is high &
- 2—the degree of the “speaker's truthfulness” is either high or low (but not medium) (i.e. when speaker's truthfulness is close to either 1 or 0, but away from 0.5) &
- 3—the degree of “expertise of the speaker” is high &
- 4—the degree of “perception of the speaker” is high
In one model, the degree of the “sureness of a speaker” of a statement is set between 0 and 1, as a normalized axis (or scale), for example, corresponding to zero (or minimum) and maximum sureness levels, respectively.
Please note that in our definitions here, there is a difference between the “sureness” and “trustworthiness” (of a speaker). For example, a speaker may have low trustworthiness, but has a high sureness. For example, for an always-liar speaker (i.e. when the speaker's degree of truthfulness is 0), the speaker has a low trustworthiness (for the listener), but has a high level of sureness. That is, for an always-liar speaker (i.e. not “trustworthy”), the conclusion from a statement becomes the reverse of the original statement, which means that the speaker has a high level of sureness (for the listener). For example, for an always-liar speaker, the statement “Event A is very rare” results in the following conclusion for the listener: “Event A is not very rare”. That is, once the listener knows (or has the knowledge) that the speaker is an always-liar speaker, the listener can still “count on” the “reverse” of the statement given by the speaker (with a high degree of “sureness”).
In another example, for a speaker that “sometimes lies” (i.e. a “sometimes-liar”, with the speaker's degree of truthfulness around 0.5), the “sureness” about the speaker is low.
(g) Broadness of a Statement:
Now, let's look at another factor, “the degree of the broadness of the statement”, with some examples. For example, in response to the question that “What is the color of the table?”, the statement “The color of the table may be green, blue, or red.” has higher degree of broadness than that of the statement “The color of the table is green.”, with respect to the information about the color of the table.
For example, in response to the question that “When does the meeting start today?”, the statement “The meeting may start in the next few hours,” has higher degree of broadness than that of the statement “The meeting starts at 10 am.”, with respect to the information about the starting time of the meeting.
In one model, the degree of the “broadness” of a statement is set between 0 and 1, as a normalized axis (or scale), for example, corresponding to zero (or minimum) and maximum (or 100 percent) broadness levels, respectively.
(h) Helpfulness of a Statement:
Now, let's look at another parameter, the degree of “helpfulness” (for a statement(for a listener or reader)), which depends on at least the following 2 parameters:
-
- 1—the degree of the “sureness of the speaker” of the statement
- 2—the degree of “broadness of the statement”
The degree of “helpfulness of a statement” is one measure of the information of a statement (for a listener or reader or the recipient of information), which is very contextual (e.g., dependent on the question asked).
For example, as shown in
-
- 1—the degree of the “sureness of the speaker” of the statement is high &
- 2—the degree of the “broadness of the statement” is low (i.e. the statement is very “specific”).
In one model, the degree of the “helpfulness” of a statement is set between 0 and 1, as a normalized axis (or scale), for example, corresponding to zero (or minimum) and maximum helpfulness levels, respectively. The degree of the “helpfulness” of a statement or information (I) is denoted by function H(I).
Please note that all the parameters above (e.g., the degree of the helpfulness) can also be expressed by percentages between 0 to 100 percent (or by any other scale, instead of scale of 0 to 1, respectively). The parameters above (e.g., the degree of the helpfulness) can be expressed by Fuzzy representations, as well.
Some Applications:
The parameters above are useful for situations that one gets input or information from one or more sources, and one wants to evaluate, filter, sort, rank, data-mine, validate, score, combine, find and remove or isolate contradictions, conclude, simplify, find and delete or isolate redundancies, criticize, analyze, summarize, or highlight a collection of multiple information pieces or data, from multiple sources with various levels of reliability, credibility, reputation, weight, risk, risk-to-benefit ratio, scoring, statistics, or past performance.
For example, these parameters are useful for editors of an article (such as Wikipedia, with various writers with various levels of credibility, knowledge, and bias), search engines in a database or on Internet (with information coming various sources, with different levels of confidence or credibility), economy or stock market prediction (based on different parameter inputs or opinions of different analysts, and various political, natural, and economical events), background check for security for people (based on multiple inputs from various sources and people, each with different credibility and security risk), medical doctors' opinions or diagnosis (based on doctors with various expertise and experience, information from various articles and books, and data from various measurements and equipment), booking flights and hotel online (with information from various web sites and travel agents, each with different reliability and confidence), an auction web site (with different seller's credibility, reliability, history, and scoring by other users), customize and purchase a computer online (with different pricing and seller's credibility, reliability, history, and scoring by other users), customer feedback (with various credibility), voting on an issue (with various bias), data mining (from various sources with different credibility and weight), and news gathering (from multiple sources of news, on TV or Internet, with various reliability and weight).
In one embodiment, an information source (S) may get its input or information from one or more other sources. In one embodiment, there is a network of other sources, connected in parallel or in series, or in combinations or mixtures of other sources in different configurations. In one embodiment, the information source S0 supplies some information to another information source S1, in a cascade of sources (with each source acting as a node in the structure), e.g., in a tree, pyramid, or hierarchical configuration (with many branches interconnected), where a listener gathers all the information from different sources and analyzes them to make a conclusion from all the information received, as shown in
Thus, the overall reliability and the overall credibility of the system (or other parameters describing the system) depends on (is a function of) the components, or the chain of sources in the relevant branch(es), going back to the source(s) of information. That is, for the overall reliability, R, we have:
R=Function (RS0, RS1, . . . RSm),
for m sources in the chain, starting from S0.
In one embodiment, for a source of information, when it comes through a cascade or chain of sources, the weakest link dominates the result. For example, the most unreliable link or source determines or dominates the overall reliability. In one embodiment, this can be modeled based on the MINIMUM function for reliability values for multiple sources. In one embodiment, this can be based on the AND function between the values. In one embodiment, this can be based on the additions on inverse values. e.g.:
(1/R)=(1/R1)+(1/R2)+ . . . +(1/RN)
(with R as the overall reliability, and RN as the reliability for source N)
In one embodiment, the sources are independent sources. In one embodiment, the sources are dependent sources (dependent on each other).
One of the advantages of the fuzzy analysis mentioned here in this disclosure is that the system can handle contradictory and duplicative information, to sort them out and make a conclusion from various inputs.
In one embodiment, the information can go through a source as a conduit, only (with no changes made on the received information by the source, itself). In another embodiment, the information can be generated, analyzed, and/or modified by the source, based on all the inputs to the source, and/or based on the source's own knowledge base (or database) and processor (or CPU, controller, analyzing module, computer, or microprocessor, to analyze, edit, modify, convert, mix, combine, conclude, summarize, or process the data).
In one embodiment, the source of information has time-dependent parameters. For example, the credibility or reliability of the source changes over time (with respect to a specific subject or all subjects). Or, the bias of the source may change for a specific topic or subject, as the time passes.
example, a news bldg, newspaper, radio show, radio host, TV show, TV news, or Internet source may have a predetermined bias or tendency toward a specific party, political idea, social agenda, or economic agenda, which may change due to the new management, owner, or host.
Search Engines and Question-Answering Systems:
Part of this section is a part of a paper by one of our inventors on the subject of search engines, titled “From search engines to question answering systems”, appeared in “Fuzzy logic and semantic web”, edited by Elie Sanchez, 2006, Elsevier B. V. publisher, Chapter 9, pages 163-210.
For one embodiment, for search engines or question-answering systems, one of the main goals is the deduction capability—the capability to synthesize an answer to a query by drawing on bodies of information which reside in various parts of the knowledge base. By definition, a question-answering system, or Q/A system for short, is a system which has deduction capability. The first obstacle is world knowledge—the knowledge which humans acquire through experience, communication and education. Simple examples are: “Icy roads are slippery,” “Princeton usually means Princeton University,” “Paris is the capital of France,” and “There are no honest politicians.” World knowledge plays a central role in search, assessment of relevance and deduction.
The problem with world knowledge is that much of it is perception-based, Perceptions—and especially perceptions of probabilities are intrinsically imprecise, reflecting the fact that human sensory organs, and ultimately the brain, have a bounded ability to resolve detail and store information. Imprecision of perceptions stands in the way of using conventional techniques—techniques which are based on bivalent logic and probability theory—to deal with perception-based information. A further complication is that much of world knowledge is negative knowledge in the sense that it relates to what is impossible and/or non-existent. For example, “A person cannot have two fathers,” and “Netherlands has no mountains.”
The second obstacle centers on the concept of relevance. There is an extensive literature on relevance, and every search engine deals with relevance in its own way, some at a high level of sophistication. There are two kinds of relevance: (a) question relevance and (b) topic relevance. Both are matters of degree. For example, on a very basic level, if the question is q: Number of cars in California? and the available information is p: Population of California is 37,000,000, then what is the degree of relevance of p to q? Another example: To what degree is a paper entitled “A New Approach to Natural Language Understanding” of relevance to the topic of machine translation.
Basically, there are two ways of approaching assessment of relevance: (a) semantic; and (b) statistical. To illustrate, in the number of cars example, relevance of p to q is a matter of semantics and world knowledge. In existing search engines, relevance is largely a matter of statistics, involving counts of links and words, with little if any consideration of semantics, Assessment of semantic relevance presents difficult problems whose solutions lie beyond the reach of bivalent logic and probability theory. What should be noted is that assessment of topic relevance is more amendable to the use of statistical techniques, which explains why existing search engines are much better at assessment of topic relevance than question relevance.
The third obstacle is deduction from perception-based information. As a basic example, assume that the question is q: What is the average height of Swedes?, and the available information is p: Most adult Swedes are tall. Another example is: Usually Robert returns from work at about 6 pm. What is the probability that Robert is home at about 6:15 pm? Neither bivalent logic nor probability theory provide effective tools for dealing with problems of this type. The difficulty is centered on deduction from premises which are both uncertain and imprecise.
Underlying the problems of world knowledge, relevance, and deduction is a very basic problem—the problem of natural language understanding. Much of world knowledge and web knowledge is expressed in a natural language. A natural language is basically a system for describing perceptions. Since perceptions are intrinsically imprecise, so are natural languages, especially in the realm of semantics.
A prerequisite to mechanization of question-answering is mechanization of natural language understanding, and a prerequisite to mechanization of natural language understanding is precisiation of meaning of concepts and proposition drawn from a natural language. To deal effectively with world knowledge, relevance, deduction and precisiation, new tools are needed. The principal new tools are: Precisiated Natural Language (PNL); Protoform Theory (PFT); and the Generalized Theory of Uncertainty (GTU). These tools are drawn from fuzzy logic a logic in which everything is, or is allowed to be, a matter of degree.
The centerpiece of new tools is the concept of a generalized constraint. The importance of the concept of a generalized constraint derives from the fact that in PNL, and GTU it serves as a basis for generalizing the universally accepted view that information is statistical in nature. More specifically, the point of departure in PNL and GTU is the fundamental premise that, in general, information is representable as a system of generalized constraints, with statistical information constituting a special case. Thus, much more general view of information is needed to deal effectively with world knowledge, relevance, deduction, precisiation and related problems. Therefore, a quantum jump in search engine IQ cannot be achieved through the use of methods based on bivalent logic and probability theory.
Deduction capability is a very important capability which the current search engines generally have not fully developed, yet. What should be noted, however, is that there are many widely used special purpose Q/A systems which have limited deduction capability. Examples of such systems are driving direction systems, reservation systems, diagnostic systems and specialized expert systems, especially in the domain of medicine.
It is of historical interest to note that question-answering systems were an object of considerable attention in the early seventies. The literature abounded with papers dealing with them. Interest in question-answering systems dwindled in the early eighties, when it became Obvious that AI was not advanced enough to provide the needed tools and technology. In recent years, significant progress toward enhancement of web intelligence has been achieved through the use of concepts and techniques related to the Semantic Web, OWL, CYC and other approaches. But such approaches, based on bivalent logic and probability theory, cannot do the job. The reason, which is not widely recognized as yet, is that bivalent logic and bivalent-logic-based probability theory have intrinsic limitations. To circumvent these limitations what are needed are new tools based on fuzzy logic and fuzzy-logic-based probability theory. What distinguishes fuzzy logic from standard logical systems is that in fuzzy logic everything is, or is allowed to be graduated, that is, be a matter of degree. Furthermore, in fuzzy logic everything is allowed to be granulated, with a granule being a clump of values drawn together by indistinguishability, similarity or proximity. It is these fundamental features of fuzzy logic that give it a far greater power to deal with problems related to web intelligence than standard tools based on bivalent logic and probability theory. An analogy to this is: In general, a valid model of a nonlinear system cannot be constructed through the use of linear components.
There are three major obstacles to upgrading a search engine to a question-answering system: (a) the problem of world knowledge; (b) the problem of relevance; and (c) the underlying problem of mechanization of natural language understanding and, in particular, the basic problem of precisiation of meaning. Since the issues to be discussed are not restricted to web-related problems, our discussion will be general in nature.
The Problem of World Knowledge:
World knowledge is the knowledge which humans acquire through experience, education and communication. Simple examples are:
-
- Few professors are rich
- There are no honest politicians
- It is not likely to rain in San Francisco in midsummer
- Most adult Swedes are tall
- There are no mountains in Holland
- Usually Princeton means Princeton University
- Paris is the capital of France
- In Europe, the child-bearing age ranges from about sixteen to about forty-two The problem with world knowledge is that much of it is perception-based. Examples:
- Most adult Swedes are tall
- Most adult Swedes are much taller than most adult Italians
- Usually a large house costs more than a small house
- There are no honest politicians
Perception-based knowledge is intrinsically imprecise, reflecting the bounded ability of sensory organs, and ultimately the brain, to resolve detail and store information. More specifically, perception-based knowledge is f-granular in the sense that (a) the boundaries of perceived classes are unsharp (fuzzy); and (b) the values of perceived attributes are imprecise (fuzzy). Bivalent-logic-based approaches provide no methods for deduction from perception-based knowledge. For example, given the datum: Most adult Swedes are tall, existing bivalent-logic-based methods cannot be employed to come up with valid answers to the questions q1: Flow many adult Swedes are short; and q2: What is the average height of adult Swedes?
The Problem of Relevance:
The importance of the concept of relevance is hard to exaggerate. Relevance is central to search. Indeed, the initial success of Google is due, in large measure, to its simple but ingenious page ranking algorithm for assessment of relevance. Despite its importance, there are no satisfactory definitions of relevance in the literature.
In fact, it may be argued that, as in the case of world knowledge, the concept of relevance is much too complex to lend itself to treatment within the limited conceptual framework of bivalent logic and bivalent-logic-based probability theory. An immediate problem is that relevance is not a bivalent concept. Relevance is a matter of degree, that is, it is a fuzzy concept. To define fuzzy concepts, what is needed is the conceptual structure of fuzzy logic. As was stated earlier, in fuzzy logic everything is, or is allowed to be, a matter of degree.
For concreteness, it is convenient to define a relevance function, R(q/p), as a function in which the first argument, q, is a question or a topic; the second argument, p, is a proposition, topic, document, web page or a collection of such objects; and R is the degree to which p is relevant to q. When q is a question, computation of R(q/p) involves an assessment of the degree of relevance of p to q, with p playing the role of question-relevant information. For example, if q: What is the number of cars in California, and p: Population of California is 37 million, then p is question-relevant to q in the sense that p constrains, albeit imprecisely, the number of cars in California. The constraint is a function of world knowledge.
If q is a topic, e.g., q: Ontology, then a document entitled p: What is ontology?, is of obvious relevance to q, i.e., p is topic-relevant. The problem in both cases is that of assessment of degree of relevance. Basically, what we need is a method of computing the degree of relevance based on the meaning of q and p, that is, we need semantic relevance. Existing search engines have a very limited capability to deal with semantic relevance. Instead, what they use is what may be called statistical relevance. In statistical relevance, what is used is, in the main, statistics of links and counts of words. Performance of statistical methods of assessment of relevance is unreliable.
A major source of difficulty in assessment of relevance relates to non-compositionality of the relevance function. More specifically, assume that we have a question, q, and two propositions p and r. Can the value of R(q/p, r) be composed from the values of R(q/p) and R(q/r)? The answer, in general, is: No. As a simple, not web-related, example, suppose that q: How old is Vera; p: Vera's age is the same as Irene's; r: Irene is 65. In this case, R(q/p)=R(q/r)=0 and yet 11(q/p, r)=1. What this implies is that, in general, relevance cannot be assessed in isolation. This suggests a need for differentiation between relevance and what may be called i-relevance, that is, relevance in isolation. In other words, a proposition, p, is i-relevant if it is relevant by itself, and it is i-irrelevant if it is not of relevance by itself, but might be relevant in combination with other propositions.
The problem of Precisiation of Meaning—A Prerequisite to Mechanization of Natural Language Understanding:
Much of world knowledge and web knowledge is expressed in a natural language. This is why issues relating to natural language understanding and natural language reasoning are of direct relevance to search and, even more so, to question-answering.
Humans have no difficulty in understanding natural language, but machines have many. One basic problem is that of imprecision of meaning. A human can understand an instruction such as “Take a few steps,” but a machine cannot. To execute this instruction, a machine needs a precisiation of “few.” Precisiation of propositions drawn from a natural language is the province of P′NL (Precisiated Natural Language). A forerunner of PNL is PRUF In PNL, precisiation is interpreted as meaning precisiation, rather than value precisiation. A proposition is precisiated through translation into the Generalized Constraint Language (GCL). An element of GCL which precisiates p is referred to as a precisiand of p, GC(p), with GC(p) representing a generalized constraint. A precisiand may be viewed as a model of meaning.
A concept which plays a key role in precisiation is cointension, with intension used in its usual logical sense as attribute-based meaning. Thus, p and q are cointensive if the meaning of p is a close approximation to that of q. In this sense, a precisiand, GC(p), is valid if GC(p) is cointensive with p. The concept of cointensive precisiation has an important implication for validity of definitions of concepts. More specifically, if C is a concept and Def(C) is its definition, then for Def(C) to be a valid definition, Def(C) must be cointensive with C (see
The concept of cointensive definition leads to an important conclusion: In general, a cointensive definition of a fuzzy concept cannot be formulated within the conceptual structure of bivalent logic and bivalent-logic-based probability theory.
See
-
- PT: standard bivalent-logic-based probability theory
- CTPM : Computational Theory of Precisiation of Meaning
- PNL: Precisiated Natural Language
- CW: Computing with Words
- GTU: Generalized Theory of Uncertainty
- GCR: Theory of Generalized-Constraint-Based Reasoning
The Concept of a Generalized Constraint:
Constraints are ubiquitous. A typical constraint is an expression of the form X∈E C, where X is the constrained variable and C is the set of values which X is allowed to take. A typical constraint is hard (inelastic) in the sense that if u is a value of X then u satisfies the constraint if and only if u∈C.
The problem with hard constraints is that most real-world constraints are not hard, meaning that most real-world constraints have some degree of elasticity. For example, the constraints “check-out time is 1 pm,” and “speed limit is 100 km/h,” are, in reality, not hard. How can such constraints be defined? The concept of a generalized constraint is motivated by questions of this kind.
Real-world constraints may assume a variety of forms. They may be simple in appearance and yet have a complex structure. Reflecting this reality, a generalized constraint, GC, is defined as an expression of the form.
GC: X isr R, where X is the constrained variable; R is a constraining relation which, in general, is nonbivalent; and r is an indexing variable which identifies the modality of the constraint, that is, its semantics. R will be referred to as a granular value of X.
The constrained variable, X, may assume a variety of forms. In particular,
-
- X is an n-ary variable, X=(X1, . . . , Xn)
- X is a proposition, e g X=Leslie is tall
- X is a function
- X is a function of another variable, X=f (Y)
- X is conditioned on another variable, X/Y
- X has a structure, e.g., X=Location(Residence(Carol))
- X is a group variable. In this case, there is a group, G[A]; with each member of the group, Namei, i=1, . . . , n, associated with an attribute-value, Ai. Ai may be vector-valued. Symbolically:
- G[A]: Name1/A1+ . . . +Namen/An.
Basically, G[A] is a relation.
-
- X is a generalized constraint, X=Y isr R.
A generalized constraint, GC, is associated with a test-score function, ts(u) which associates with each object, u, to which the constraint is applicable, the degree to which u satisfies the constraint. Usually, ts(u) is a point in the unit interval. However, if necessary, the test-score may be a vector, an element of a semi-ring, an element of a lattice or, more generally, an element of a partially ordered set, or a bimodal distribution. The test-score function defines the semantics of the constraint with which it is associated.
The constraining relation, R, is, or is allowed to be, non-bivalent (fuzzy). The principal modalities of generalized constraints are summarized in the following.
Principal Modalities of Generalized Constraints:
(a) Possibilistic (r=blank)
X is It
with R playing the role of the possibility distribution of X. For example:
X is [a, b]
means that [a, b] is the set of possible values of X. Another example:
X is small.
In this case, the fuzzy set labeled small is the possibility distribution of X. If μsmall is the membership function of small, then the semantics of “X is small” is defined by
Poss{X=u}=μsmall(u)
where u is a generic value of X.
(b) Probabilistic (r=p)
X isp R,
with R playing the role of the probability distribution of X. For example:
X isp N(m, σ2) means that X is a normally distributed random variable with mean m and variance σ2.
If X is a random variable which takes values in a finite set {u1, . . . , un} with respective probabilities p1, . . . , pn, then X may be expressed symbolically as
X isp (p1\u1+ . . . +pn\un),
with the semantics
Prob(X=ui)=pi, (i=1, . . . n).
What is important to note is that in the Generalized Theory of Uncertainty (GTU), a probabilistic constraint is viewed as an instance of a generalized constraint.
When X is a generalized constraint, the expression
X isp R
is interpreted as a probability qualification of X, with R being the probability of X. For example:
(X is small) isp likely,
where small is a fuzzy subset of the real line, means that the probability of the fuzzy event {X is small} is likely. More specifically, if X takes values in the interval [a, b] and g is the probability density function of X, then the probability of the fuzzy event “X is small” may be expressed as the following integral, taken between a and b interval:
Hence:
This expression for the test-score function defines the semantics of probability qualification of a possibillistic constraint.
(c) Veristic (r=v)
X isv R,
where R plays the role of a verity (truth) distribution of X. In particular, if X takes values in a finite set {u1, . . . , un} with respective verity (truth) values t1, . . . , tn, then X may be expressed as
X isv (t1|u1+ . . . +tn|un),
meaning that Ver(X=ui)=ti, i=1, . . . , n.
For example, if Robert is half German, quarter French and quarter an, then
Ethnicity(Robert) isv (0.5 German +0.25 French +0.25 Italian).
When X is a generalized constraint, the expression
X isv R
is interpreted as verity (truth) qualification of X. For example, (X is small) isv very true,
should be interpreted as “It is very true that X is small.” The semantics of truth qualification is defined this way.
Ver(X is R) is t→X is μR−1 (t),
where μR−1 is inverse of the membership function of R and t is a fuzzy truth value which is a subset of [0, 1], as shown in
Note. There are two classes of fuzzy sets: (a) possibilistic, and (b) veristic. In the case of a possibilistic fuzzy set, the grade of membership is the degree of possibility. In the case of a veristic fuzzy set, the grade of membership is the degree of verity (truth). Unless stated to the contrary, a fuzzy set is assumed to be possibilistic.
(d) Usuality (r=u)
X isu R.
The usuality constraint presupposes that X is a random variable, and that probability of the event {X isu R} is usually, where usually plays the role of a fuzzy probability which is a fuzzy number. For example:
X isu small
means that “usually X is small” or, equivalently,
Prob {X is small} is usually.
In this expression, small may be interpreted as the usual value of X. The concept of a usual value has the potential of playing a significant role in decision analysis, since it is more informative than the concept of expected value.
(e) Random-set (r=rs)
In
X isrs R,
X is a fuzzy-set-valued random variable and R is a fuzzy random set.
(f) Fuzzy-graph (r=fg)
In
X isfg R,
X is a function, f, and R is a fuzzy graph which constrains f (see
R=A1×B1+ . . . +An×Bn,
where the Ai and Bi , i=1, . . . , n, are fuzzy subsets of the real line, and × is the Cartesian product. A fuzzy graph is frequently described as a collection of fuzzy if-then rules.
R: if X is A1 then Y is B1, i=1, . . . , n.
The concept of a fuzzy-graph constraint plays an important role in applications of fuzzy logic.
(g) Bimodal (r=bm)
In the bimodal constraint,
X isbm R,
R is a bimodal distribution of the form
R: ΣiPi\Ai, i=1, . . . , n,
which means that Prob(X is Ai) is Pi.
To clarify the meaning of a bimodal distribution, it is expedient to start with an example. I am considering buying Ford stock. I ask my stockbroker, “What is your perception of the near-term prospects for Ford stock?” He tells me, “A moderate decline is very likely; a steep decline is unlikely; and a moderate gain is not likely.” My question is: What is the probability of a large gain?
Information provided by my stock broker may be represented as a collection of ordered pairs:
Price: ((unlikely, steep.decline), (very likely, moderate.decline), (not likely, moderate.gain)).
In this collection, the second element of an ordered pair is a fuzzy event or, generally, a possibility distribution, and the first element is a fuzzy probability. The expression for Price is an example of a bimodal distribution.
The importance of the concept of a bimodal distribution derives from the fact that in the context of human-centric systems, most probability distributions are bimodal. Bimodal distributions can assume a variety of forms. The principal types are Type 1, Type 2 and Type 3. Type 1, 2 and 3 bimodal distributions have a common framework but differ in important detail. A bimodal distribution may be viewed as an important generalization of standard probability distribution. For this reason, bimodal distributions of Type 1, 2, 3 are discussed in greater detail in the following.
-
- Type 1 (default): X is a random variable taking values in U
A1, . . . , An, A are events (fuzzy sets)
pi=Prob(X is Ai), Prob(X is Ai) is Pi , i=1, . . . , n,
Σi pi is unconstrained
BD: bimodal distribution: ((P1,A1), . . . , (Pn,An))
or, equivalently,
X isbm (P1 \A1+ . . . +Pn\An)
Now, what is the probability, p, of A? In general, this probability is fuzzy-set-valued.
A special case of bimodal distribution of Type 1 is the basic bimodal distribution (BBD). In BBD, X is a real-valued random variable, and X and P are granular. (See
-
- Type 2 (fuzzy random set): X is a fuzzy-set-valued random variable with values
- A1, . . . , An (fuzzy sets)
- pi=Prob(X=Ai), Prob(X is Ai) is Pi , i=1, . . . , n
BD: X isrs (P1\A1+ . . . +Pn\An)
Σi Pi=1,
where the Pi are granular probabilities.
Now, what is the probability, P, of A? P is not definable. What are definable are (a) the expected value of the conditional possibility of A given BD, and (b) the expected value of the conditional necessity of A given BD.
-
- Type 3 (Dempster-Shafer): X is a random variable taking values X1, . . . , Xn with probabilities p1, . . . , pn.
Xi is a random variable taking values in Ai, i=1, . . . , n
Probability distribution of Xi in Ai, i=1, . . . , n, is not specified.
Now, what is the probability, p, that X is in A? Because probability distributions of the Xi in the Ai are not specified, p is interval-valued. What is important to note is that the concepts of upper and lower probabilities break down when the Ai are fuzzy sets,
Note: In applying Dempster-Shafer theory, it is important to check on whether the data fit Type 3 model. In many cases, the correct model is Type 1 rather than Type 3.
The importance of bimodal distributions derives from the fact that in many realistic settings a bimodal distribution is the best approximation to our state of knowledge. An example is assessment of degree of relevance, since relevance is generally not well defined. If I am asked to assess the degree of relevance of a book on knowledge representation to summarization, my state of knowledge about the book may not be sufficient to justify an answer such as 0.7. A better approximation to my state of knowledge may be “likely to be high.” Such an answer is an instance of a bimodal distribution.
(h) Group (r=g)
In
X isg R,
X is a group variable, G[A], and R is a group constraint on G[A]. More specifically, if X is a group variable of the form
G[A]: Name1/A1+ . . . +Namen/An
or
G[A]: Σi Namei/Ai, for short, i=1, . . . , n,
then R is a constraint on the Ai. To illustrate, if we have a group of n Swedes, with Namei being the name of i-th Swede, and Ai being the height of Name; , then the proposition “most Swedes are tall,” is a constraint on the Ai which may be expressed as:
(1/n) ΣCount(tall.Swedes) is most
or, more explicitly,
(1/n) (μtall(A1)+ . . . +μtall(An) is most,
where most is a fuzzy quantifier which is interpreted as a fuzzy number.
Operations on Generalized Constraints:
There are many ways in which generalized constraints may be operated on. The basic operations—expressed in symbolic form—are the following.
(a) Conjunction
X isr R
Y iss S
(X, Y) ist T
EXAMPLE (possibilistic constraints).
X is R
Y is S
(X, Y) is R×S
where × is the Cartesian product.
EXAMPLE (probabilistic/possibilistic).
X isp R
(X, Y) is S
(X, Y) isrs T
In this example, if S is a fuzzy relation then T is a fuzzy random set. What is involved in this example is a conjunction of a probabilistic constraint and a possibilistic constraint. This type of probabilistic/possibilistic constraint plays a key role in the Dempster-Shafer theory of evidence, and in its extension to fuzzy sets and fuzzy probabilities.
EXAMPLE (possibilistic/probabilistic).
X is R
(X, Y) isp S
Y/X isp T
This example, which is a dual of the proceeding example, is an instance of conditioning.
(b) Projection (possibilistic)
(X, Y) is R
X is S
where X takes values in U={u}; Y takes values in V={v}; and the projection
S=Proj×R,
is defined as
μS(u)=μProj x R(u)=maxv μR(u, v),
where μR and μS are the membership functions of R and S, respectively.
(c) Projection (probabilistic)
(X, Y) isp R
X isp S
where X and Y are real-valued random variables, and R and S are the probability distributions of (X, Y) and X, respectively. The probability density function of S, pS, is related to that of R, pR, by the familiar equation
pS(u)=∫ pR(u, v) dv
with the integral taken over the real line.
(d) Propagation
f(X) isr R
g(X) iss S
where f and g are functions or functionals.
EXAMPLE (possibilistic constraints).
f(X) is R
g(X) is S
where R and S are fuzzy sets. In terms of the membership function of R, the membership function of S is given by the solution of the variational problem
μS(v)=sup u(μRf(u))
subject to
v=g(u).
Note. The constraint propagation rule described in this example is the well-known extension principle of fuzzy logic. Basically, this principle provides a way of computing the possibilistic constraint on g(X) given a possibilistic constraint on f(X).
See
f(X) is A
g(X) is B
μS(v)=supu(μA(f(u)))
subject to
v=g(u).
Primary constraints, composite constraints and standard constraints:
Among the principal generalized constraints there are three that play the role of primary generalized constraints. They are:
Possibilistic constraint: X is R
Probabilistic constraint: X isp R
and
Veristic constraint: X isv R
A special case of primary constraints is what may be called standard constraints: bivalent possibilistic, probabilistic and bivalent veristic. Standard constraints form the basis for the conceptual framework of bivalent logic and probability theory. A generalized constraint, GC, is composite if it can be generated from other generalized constraints through conjunction, and/or projection, and/or constraint propagation, and/or qualification and/or possibly other operations. For example, a random-set constraint may be viewed as a conjunction of a probabilistic constraint and either a possibilistic or veristic constraint. The Dempster-Shafer theory of evidence is, in effect, a theory of possibilistic random-set constraints. The derivation graph of a composite constraint defines how it can be derived from primary constraints.
The three primary constraints—possibilistic, probabilistic and veristic—are closely related to a concept which has a position of centrality in human cognition—the concept of partiality. In the sense used here, partial means: a matter of degree or, more or less equivalently, fuzzy. In this sense, almost all human concepts are partial (fuzzy). Familiar examples of fuzzy concepts are: knowledge, understanding, friendship, love, beauty, intelligence, belief, causality, relevance, honesty, mountain and, most important, truth, likelihood and possibility. Is a specified concept, C, fuzzy? A simple test is: If C can be hedged, then it is fuzzy. For example, in the case of relevance, we can say: very relevant, quite relevant, slightly relevant, etc. Consequently, relevance is a fuzzy concept.
The three primary constraints may be likened to the three primary colors: red, blue and green. In terms of this analogy, existing theories of uncertainty may be viewed as theories of different mixtures of primary constraints. For example, the Dempster-Shafer theory of evidence is a theory of a mixture of probabilistic and possibilistic constraints. The Generalized Theory of Uncertainty (GTU) embraces all possible mixtures. In this sense the conceptual structure of GTU accommodates most, and perhaps all, of the existing theories of uncertainty.
The Generalized Constraint Language and Standard Constraint Language:
A concept which has a position of centrality in PNL is that of Generalized Constraint Language (GCL). Informally, GCL is the set of all generalized constraints together with the rules governing syntax, semantics and generation. Simple examples of elements of GCL, are:
((X, Y) isp A)∧(X is B)
(X isp A)∧((X, Y) isv B)
Proj Y ((X is A)∧((X, Y) isp B)),
where A is conjunction.
A very simple example of a semantic rule is:
(X is A)∧(Y is B)→Poss(X=u, Y=v)=μA(u)∧μB(v),
where u and v are generic values of X, Y , and μA and μB are the membership functions of A and B, respectively.
In principle, GCL is an infinite set. However, in most applications only a small subset of GCL is likely to be needed,
In PNL, the set of all standard constraints together with the rules governing syntax, semantics and generation constitute the Standard Constraint Language (SCL). SCL is a subset of GCL.
The Concept of Cointensive Precisiation:
As was pointed out already, much of world knowledge and web knowledge is expressed in a natural language. For this reason, mechanization of natural language understanding is of direct relevance to enhancement of web intelligence. In recent years, considerable progress has been made in areas of computational linguistics which relate to mechanization of natural language understanding. But what is widely unrecognized is that there is a fundamental limitation to what can be achieved through the use of commonly-employed methods of meaning representation. The aim of what follows is, first, to highlight this limitation and, second, to present ways of removing it.
To understand the nature of the limitation, two facts have to be considered. First, as was pointed out earlier, a natural language, NL, is basically a system for describing perceptions; and second, perceptions are intrinsically imprecise, reflecting the bounded ability of human sensory organs, and ultimately the brain, to resolve detail and store information. A direct consequence of imprecision of perceptions is semantic imprecision of natural languages. Semantic imprecision of natural languages is not a problem for humans, but is a major problem for machines.
To clarify the issue, let p be a proposition, concept, question or command. For p to be understood by a machine, it must be precisiated, that is, expressed in a mathematically well-defined language. A precisiated form of p, Pre(p), will be referred to as a precisiand of p and will be denoted as p*. The object of precisiation, p, will be referred to us precisiend.
To precisiate p we can employ a number of meaning-representation languages, e.g., Prolog, predicate logic, semantic networks, conceptual graphs, LISP, SQL, etc. The commonly-used meaning-representation languages are bivalent, i.e., are based on bivalent logic. Are we moving in the right direction when we employ such languages for mechanization of natural language understanding? The answer is: No. The reason relates to an important issue which we have not addressed: cointension of p*, with intension used in its logical sense as attribute-based meaning. More specifically, cointension is a measure of the goodness of fit of the intension of a precisiand, p*, to the intended intension of precisiend, p. Thus, cointension is a desideratum of precisiation. What this implies is that mechanization of natural language understanding requires more than precisiation—a it requires cointensive precisiation. Note that definition is a form of precisiation. In plain words, a definition is cointensive if its meaning is a good fit to the intended meaning of the definiendum.
Here is where the fundamental limitation which was alluded to earlier comes into view. In a natural language, NL, most p's are fuzzy, that is, are in one way or another, a matter of degree. Simple examples: propositions “most Swedes are tall” and “overeating causes obesity;” concepts “mountain” and “honest;” question “is Albert honest?” and command “take a few steps.”
Employment of commonly-used meaning-representation languages to precisiate a fuzzy p leads to a bivalent (crisp) precisiend p*. The problem is that, in general, a bivalent p* is not cointensive. As a simple illustration, consider the concept of recession. The standard definition of recession is: A period of general economic decline; specifically, a decline in GDP for two or more consecutive quarters. Similarly, a definition of bear market is: We classify a bear market as a 30 percent decline after 50 days, or a 13 percent decline after 145 days. (Robert Shuster, Ned Davis Research.) Clearly, neither definition is cointensive.
Another example is the classical definition of stability. Consider a ball of diameter D which is placed on an open bottle whose mouth is of diameter d. If D is somewhat larger than d, the configuration is stable: Obviously, as D increases, the configuration becomes less and less stable. But, according to Lyapounov's bivalent definition of stability, the configuration is stable for all values of D greater than d. This contradiction is characteristic of crisp definitions of fuzzy concepts—a well-known example of which is the Greek sorites (heap) paradox.
The magnitude of the problem becomes apparent when we consider that many concepts in scientific theories are fuzzy, but are defined and treated as if they are crisp. This is particularly true in fields in which the concepts which are defined are descriptions of perceptions. To remove the fundamental limitation, bivalence must be abandoned. Furthermore, new concepts, ideas and tools must be developed and deployed to deal with the issues of cointensive precisiation, definability and deduction. The principal tools are Precisiated Natural Language (PNL); Protoform Theory (PFT); and the Generalized Theory of Uncertainty (GTU). These tools form the core of what may be called the Computational Theory of Precisiation of Meaning (CTPM). The centerpiece of CTPM is the concept of a generalized constraint.
The concept of a generalized constraint plays a key role in CTPM by providing a basis for precisiation of meaning. More specifically, if p is a proposition or a concept, its precisiand, Pre(p), is represented as a generalized constraint, GC. Thus, Pre(p)=GC. In this sense, the concept of a generalized constraint may be viewed as a bridge from natural languages to mathematics.
See
Annotated translation:
p→X/A isr R/B←GC(p)
Representing precisiands of p as elements of GCL is the pivotal idea in CTPM. Each precisiand is associated with the degree to which it is cointensive with p. Given p, the problem is that of finding those precisiands which are cointensive, that is, have a high degree of cointension. If p is a fuzzy proposition or concept, then in general there are no cointensive precisiands in SCL.
In CTPM, a refinement of the concept of precisiation is needed. First, a differentiation is made between v-precision (precision in value) and m-precision (precision in meaning). For example, proposition p: X is 5, is both v-precise and m-precise; p: X is between 5 and 7, is v-imprecise and m-precise; and p: X is small, is both v-imprecise and m-imprecise; however, p can be m-precisiated by defining small as a fuzzy set or a probability distribution. A perception is v-imprecise and its description is m-imprecise. PNL makes it possible to m-precisiate descriptions of perceptions.
Granulation of a variable, e.g., representing the values of age as young, middle-aged and old, may be viewed as a form of v-imprecisiation. Granulation plays an important role in human cognition by serving as a means of (a) exploiting a tolerance for imprecision through omission of irrelevant information; (b) lowering precision and thereby lowering cost; and (c) facilitating understanding and articulation. In fuzzy logic, granulation is m-precisiated through the use of the concept of a linguistic variable. Further refinement of the concept of precisiation relates to two modalities of m-precisiation: (a) human-oriented, denoted as mh-precisiation; and (b) machine-oriented, denoted as mm-precisiation. Unless stated to the contrary, in CTPM, precisiation should be understood as mm-precisiation. (See
In a bimodal dictionary or lexicon, the first entry, p, is a concept or proposition; the second entry, p*, is mh-precisiand of p; and the third entry is mm-precisiand of p. To illustrate, the entries for recession might read: mh-precisiand a period of general economic decline; and mm-precisiand—a decline in GDP for two or more consecutive quarters. (See
There is a simple analogy which helps to understand the meaning of cointensive precisiation. Specifically, a proposition, p, is analogous to a system, S; precisiation is analogous to triodelization; a precisiand, expressed as a generalized constraint, GC(p), is analogous to a model, M(S), of S; test-score function is analogous to input-output relation; cointensive precisiand is analogous to well-fitting model; GCL is analogous to the class of all fuzzy-logic-based systems; and SCL is analogous to the subclass of all bivalent-logic-based systems. To say that, in general, a cointensive definition of a fuzzy concept cannot be formulated within the conceptual structure of bivalent logic and probability theory, is similar to saying that, in general, a linear system cannot be a well-fitting model of a nonlinear system,
See
input-output relation→intension
degree of match between M(S) and S→cointension
Ramifications of the concept of cointensive precisiation extend well beyond mechanization of natural language understanding. A broader basic issue is validity of definitions in scientific theories, especially in the realms of human-oriented fields such as law, economics, medicine, psychology and linguistics. More specifically, the concept of cointensive precisiation calls into question the validity of many of the existing definitions of basic concepts among them the concepts of causality, relevance, independence, stability, complexity, and optimality.
Translation of p into GCL is made more transparent though annotation. To illustrate,
(a) p: Monika is young→X/Age(Monika) is R/young.
(b) p: It is like that Monika is young→Prob(X/Age(Monika) is R/young) is S/likely
Note: Example (b) is an instance of probability qualification.
More concretely, let g(u) be the probability density function of the random variable, Age(Monika). Then, with reference to our earlier discussion of probability qualification, we have:
is likely, or, in annotated form,
The test-score of this constraint on g is given by
(c) p: Most Swedes are tall.
Following (b), let h(u) be the count density function of Swedes, meaning that h(u) du=fraction of Swedes whose height lies in the interval [u, u+du]. Assume that height of Swedes lies in the interval [a, b]. Then,
fraction of tall Swedes:
Interpreting this relation as a generalized constraint on h, the test-score may be expressed as:
In summary, precisiation of “Most Swedes are tall” may be expressed as the generalized constraint.
An important application of the concept of precisiation relates to precisiation of propositions of the form “X is approximately a,” where a is a real number. How can “approximately a,” or *a (for short), be precisiated? In other words, how can the uncertainty associated with the value of X which is described as *a, be defined precisely? There is a hierarchy of ways in which this can be done. The simplest is to define *a as a. This mode of precisiation will be referred to as singular precisiation, or s-precisiation, for short. s-precisiation is employed very widely, especially in probabilistic computations in which an imprecise probability, *a, is computed with as if it were an exact number, a.
The other ways will be referred to as granular precisiation, or g-precisiation, for short. In g-precisiation, *a is treated as a granule. What we see is that various modes of precisiating *a are instances of the generalized constraint. The concept of precisiation has an inverse the concept of imprecisiation, which involves replacing a with *a, with the understanding that *a is not unique. Imprecisiation has a negative connotation. In fact, imprecisiation serves an important purpose. More specifically, consider a proposition p of the form
p: X is V,
where X is a variable and V is its value. X may assume a variety of forms. In particular, X may be a real-valued variable, an n-ary variable, a function or a relation. The value, V, is v-precise if it is singular, that is, V is a singleton. V is v-imprecise if it is granular. In this framework, v-imprecisiation may be interpreted as a transition from singular to granular value of V .
v-imprecisiation is forced (necessary) when the value of V is not known precisely, v-imprecisiation is deliberate (optional) if there is no need for V to be known precisely. In this case, what may be called v-imprecisiation principle comes into play.
v-imprecisiation principle: Precision carries a cost. If there is a tolerance for imprecision, exploit it by employing v-imprecisiation to achieve lower cost, robustness, tractability, decision-relevance and higher level of confidence.
A word about confidence: If V is uncertain, the confidence in p, Con(p), may be defined as the probability that p is true. Generally, v-imprecisiation of V serves to increase Con(p). For example, Con(Carol is young)>Con(Carol is 23). Thus, as a rule, confidence increases when specificity decreases.
An important example is granulation. In fuzzy logic, granulation may be interpreted as v-imprecisiation followed by mm-precisiation. In this perspective, the concept of granulation—in combination with the associated concept of a linguistic variable may be viewed as one of the major contributions of fuzzy logic.
A basic problem which relates to imprecisiation is the following. Assume for simplicity that we have two linear equations involving real-valued coefficients and real-valued variables:
a11X+a12Y=b1,
a21X+a22Y=b2.
Solutions of these equations read,
X=((a22b1−a12b2)/(a11a22−a12a21)),
Y=((a11b2−a21b1)/(a11a22−a12a21)).
Now suppose that we imprecisiate the coefficients, replacing, aij with *aij, i, j=1, 2, and replacing bi with *bi, i=1, 2. How can we solve these equations when imprecisiated coefficients are defined as generalized constraints?
There is no general answer to this question. Assuming that all coefficients are defined in the same way, the method of solution will depend on the modality of the constraint. For example, if the coefficients are interval-valued, the problem falls within the province of interval analysis. If the coefficients are fuzzy-interval-valued, the problem falls within the province of the theory of relational equations. And if the coefficients are real-valued random variables, we are dealing with the problem of solution of stochastic equations.
One complication is the following. If (a) we solve the original equations, as we have done above; (b) imprecisiate the coefficients in the solution; and (c) employ the extension principle to complete X and Y, will we obtain solutions of imprecisiated equations? The answer, in general, is: No.
Nevertheless, when we are faced with a problem which we do not know how to solve correctly, we proceed as if the answer is: Yes. This common practice may be described as Precisiation/Imprecisiation Principle which is defined in the following.
Precisiation/Imprecisiation Principle (P/I Principle):
Informally, let f be a function or a functional. Y=f(X), where X and Y are assumed to be imprecise, Pr(X) and Pr(Y) are precisiations of X and Y, and *Pr(X) and *Pr(Y) are imprecisiations of Pr(X) and Pr(Y); respectively. In symbolic form, the P/I principle may be expressed as
f(X)*=*f(Pr(X)),
where *=denotes “approximately equal,” and *f is imprecisiation of f . In words, to compute f(X) when X is imprecise, (a) precisiate X, (b) compute f(Pr(X)); and (c) imprecisiate f(Pr(X)). Then, usually, *f (Pr(X)) will be approximately equal to f(X). An underlying assumption is that approximations are commensurate in the sense that the closer Pr(X) is to X, the closer f (Pr(X)) is to f(X). This assumption is related to the concept of gradual rules of Dubois and Prade.
As an example, suppose that X is a real-valued function; f is the operation of differentiation, and *X is the fuzzy graph of X. Then, using the P/I principle, *f(X) is obtained. It should be underscored that imprecisiation is an imprecise concept.
Use of the P/I principle underlies many computations in science, engineering, economics and other fields. In particular, as was alluded to earlier, this applies to many computations in probability theory which involve imprecise probabilities. It should be emphasized that the P/I principle is neither normative (prescriptive) nor precise; it merely describes imprecisely what is common practice—without suggesting that common practice is correct.
Precisiation of Propositions:
In preceding discussion, we focused our attention on precisiation of propositions of the special form “X is *a.” In the following, we shall consider precisiation in a more general setting. In this setting, the concept of precisiation in PNL opens the door to a wide-ranging enlargement of the role of natural languages in scientific theories, especially in fields such as economics, law and decision analysis.
Within CTPM, precisiation of propositions—and the related issues of precisiation of questions, commands and concepts—falls within the province of PNL. As was stated earlier, the point of departure in PNL is representation of a precisiand of a proposition, p, as a generalized constraint.
p→X isr R.
To illustrate precisiation of propositions and questions, it will be useful to consider some examples.
(a) The Robert example:
p: Usually Robert returns from work at about 6 pm.
Q: What is the probability that Robert is home at about 6:15 pm?
Precisiation of p may be expressed as
p: Prob(Time(Return(Robert)) is *6:00 pm) is usually
where “usually” is a fuzzy probability.
Assuming that Robert stays home after returning from work, precisiation of q may be expressed as
q: Prob(Time(Return(Robert)) is≤∘6:15 pm) is A?
where ∘ is the operation of composition, and A is a fuzzy probability.
(b) The balls-in-box problem:
p1: A box contains about 20 black and white balls
p2: Most are black
p3: There are several times as many black balls as white balls
q1: What is the number of white balls?
q2: What is the probability that a ball drawn at random is white?
Let X be the number of black balls and let Y be the number of white balls. Then, in precisiated form, the statement of the problem may be expressed as:
For the data, we have:
p1: (X+Y) is *20
p2: X is most ×*20
p3: X is several ×Y,
And, for the questions, we have:
q1: Y is?A
q2: Y/*20 is ?B,
where Y/*20 is the granular probability that a ball drawn at random is white.
Solution of these equations reduces to an application of fuzzy integer programming. (See
(c) The tall Swedes problem:
p: Most Swedes are tall.
Q: What is the average height of Swedes?
Q: How many Swedes are short?
As was shown earlier,
p:
is most,
where h is the count density function.
Precisiations of q1 and q2 may be expressed as
q1:
where A is a fuzzy number which represents the average height of Swedes, and
q2:
where μshort is the membership function of short, and B is the fraction of short Swedes,
(d) The partial existence problem:
X is a real number. I am uncertain about the value of X. What I know about X is:
p1: X is much larger than approximately a,
p2: X is much smaller than approximately b,
where a and b are real numbers, with a<b.
What is the value of X?
In this case, precisiations of data may be expressed as
p1: X is much larger ∘*a
p2: X is much smaller ∘*b,
where is the operation of composition. Precisiation of the question is:
q: X is ?A,
where A is a fuzzy number. The solution is immediate:
X is (much.larger ∘*a∧much.smaller ∘*b),
when ∧ is min or a t-norm. In this instance, depending on a and b, X may exist to a degree.
These examples point to an important aspect of precisiation. Specifically, to precisiate p, we have to precisiate or, equivalently, calibrate its lexical constituents. For example, in the case of “Most Swedes are tall,” we have to calibrate “most” and “tall.” Likewise, in the case of the Robert example, we have to calibrate “about 6:00 pm,” “about 6:15 pm” and “usually.” In effect, we are composing the meaning of p from the meaning of its constituents. This process is in the spirit of Frege's principle of compositionality, Montague grammar and the semantics of programming languages.
In probability theory, for example, independence of events is a bivalent concept. But, in reality, independence is a matter of degree, i.e., is a fuzzy concept. PNL, used as a definition language, makes it possible, more realistically, to define independence and other bivalent concepts in probability theory as fuzzy concepts. For this purpose, when PNL is used as a definition language, a concept is first defined in a natural language and then its definition is precisiated through the use of PNL.
The Concept of a Protoform:
Viewed in a broader perspective, what should be noted is that precisiation of meaning is not the ultimate goal—it is an intermediate goal. Once precisiation of meaning is achieved, the next goal is that of deduction from decision-relevant information. The ultimate goal is decision.
In CTPM, a concept which plays a key role in deduction is that of a protoform—an abbreviation for prototypical form. Informally, a protoform of an object is its abstracted summary. More specifically, a protoform is a symbolic expression which defines the deep semantic structure of an object such as a proposition, question, command, concept, scenario, or a system of such objects. In the following, our attention will be focused on protoforms of propositions, with PF(p) denoting a protoform of p. Abstraction has levels, just as summarization does. For this reason, an object may have a multiplicity of protoforms. Conversely, many objects may have the same protoform. Such objects are said to be protoform-equivalent, or PF-equivalent, for short. The set of protoforms of all precisiable propositions in NL, together with rules which govern propagation of generalized constraints, constitute what is called the Protoform Language (PFL). (See
EXAMPLES
Monika is young→Age(Monika) is young→A(B) is C,
where Age refers to A, Monika to B (as instantiation), and Young to C (as abstraction)
Monika is much younger than Pat→(A(B), A(C)) is R,
where Age refers to A, Monika to B, Pat to C, and “much younger” to R.
distance between New York and Boston is about 200 mi→A(B,C) is R,
where Distance refers to A, New York to B, Boston to C, and “about 200 miles” to D.
usually Robert returns from work at about 6 pm→Prob{A is B} is C,
where “Time(Robert.returnsfrom.work)” refers to A, “about 6 pm” to B, and Usually to C.
Carol lives in a small city near San Francisco→A(B(C)) is (D and E),
where “small city” refers to E, “city near SF” to D, Carol to C, Residence to B, and Location to A.
most Swedes are tall→1/n ΣCount(G[A] is R) is Q,
where Most refers to Q, Swedes to G, tall to R, and Height to A.
Another example: Alan has severe back pain. He goes to see a doctor. The doctor tells him that there are two options: (1) do nothing; and (2) do surgery. In the case of surgery, there are two possibilities: (a) surgery is successful, in which case, Alan will be pain free; and (b) surgery is not successful, in which case Alan will be paralyzed from the neck down. (See
Protoformal Deduction:
The rules of deduction in CTPM are, basically, the rules which govern constraint propagation. In CTPM, such rules reside in the Deduction Database (DDB). The Deduction Database comprises a collection of agent-controlled modules and submodules, each of which contains rules drawn from various fields and various modalities of generalized constraints. A typical rule has a symbolic part, which is expressed in terms of protoforms; and a computational part which defines the computation that has to be carried out to arrive at a conclusion.
See
-
- in PNL, deduction=generalized constraint propagation
- PFL: Protoform Language
- DDB: deduction database=collection of protoformal rules governing generalized constraint propagation
- WKDB: World Knowledge Database (PNL-based)
See also
(a) Computational rule of inference:
For symbolic part, we have:
X is A
(X, Y) is B
Y is C
For computational part, we have:
μC(v)=maxu(μA(u) ∧μB(u, v))
(b) Intersection/product syllogism:
For symbolic part, we have:
Q1 A's are B's
Q2 (A&B)'s are C's
Q3 A's are (B&C)'s
For computational part, we have:
Q3=Q1*Q2
where Q1 and Q2 are fuzzy quantifiers; A,B,C are fuzzy sets; * is product in fuzzy arithmetic.
(c) Basic extension principle:
For symbolic part, we have:
X is A
f(X) is B
For computational part, we have:
μB(v)=supn(μA(u))
subject to
v=f(u)
g is a given function or functional; A and B are fuzzy sets.
(d) Extension principle:
This is the principal rule governing possibilistic constraint propagation.
For symbolic part, we have:
f(X) is A
g(X) is B
For computational part, we have:
μB(v)=supu(μB(f(u)))
subject to
v=g(u)
Note. The extension principle is a primary deduction rule in the sense that many other deduction rules are derivable from the extension principle. An example is the following rule.
(e) Basic probability rule:
For symbolic part, we have:
Prob(X is A) is B
Prob(X is C) is D
For computational part, we have:
X is a real-valued random variable; A, B, C, and D are fuzzy sets: r is the probability density of X; and U={u}. To derive this rule, we note that
which are generalized constraints of the form
f(r) is B
g(r) is D.
Applying the extension principle to these expressions, we obtain the expression for D which appears in the basic probability rule.
(f) Bimodal interpolation rule:
The bimodal interpolation rule is a rule which resides in the Probability module of DDB. The symbolic and computational parts of this rule are:
Symbolic parts:
Prob(X is Ai) is Pi
Prob(X is A) is Q
where i=1, . . . , n
Computational parts:
In this rule, X is a real-valued random variable; r is the probability density of X; and U is the domain of X,
Note: The probability rule is a special case of the bimodal interpolation rule.
What is the expected value, E(X), of a bimodal distribution? The answer follows through application of the extension principle:
Note. E(X) is a fuzzy subset of U.
(g) Fuzzy-graph interpolation rule:
This rule is the most widely used rule in applications of fuzzy logic. We have a function, Y=f(X), which is represented as a fuzzy graph. The question is: What is the value of Y when X is A? The Ai , Bi and A are fuzzy sets.
Symbolic part is:
X is A
Y=f(X)
f(X) isfg Σi Ai×Bi
Y is C
Computational part is:
C=Σimi∧Bi,
where mi is the degree to which A matches Ai
mi=supu(μA(u) ∧μAi(u)),
When A is a singleton, this rule reduces to
X=a
Y=f(X)
f(X) isfg Σi Ai×Bi
i=1, . . . , n.
YRAJ (a) A B.
In this form, the fuzzy-graph interpolation rule coincides with the Mamdani rule—a rule is widely used in control and related applications.
In the foregoing, we have summarized some of the basic rules in DDB which govern generalized constraint propagation. A few examples of such rules are the following.
(a) Probabilistic extension principle:
f(X) isp A
g(X) isr ?B
(b) Usuality-qualified extension principle:
f(X) isu A
g(X) isr ?B
(c) Usuality-qualified fuzzy-graph interpolation rule:
X is A
Y=f(X)
f(X) isfg Σi if X is Ai then Y isu Bi
Y isr ?B
(d) Bimodal extension principle:
X isbm ρi Pi\Ai
Y=f(X)
Y isr ?B
(e) Bimodal, binary extension principle:
X isr R
Y iss S
Z=f (X, Y)
Z ist T
In the instance, bimodality means that X and Y have different modalities, and binary means that f is a function of two variables. An interesting special case is one in which X is R and Y isp S.
The deduction rules which were briefly described in the foregoing are intended to serve as examples:
(a) The Robert example:
p: Usually Robert returns from work at about 6:00 pm. What is the probability that Robert is home at about 6:15 pm?
First, we find the protoforms of the data and the query.
Usually Robert returns from work at about 6:00 pm
→Prob(Time(Return(Robert)) is *6:00 pm) is usually
which in annotated form reads
→Prob(X/Time(Return(Robert)) is A/*6:00 pm) is D/usually.
Likewise, for the query, we have
Prob(Time(Return(Robert)) is ≤∘*6:15 pm) is ?D
which in annotated form reads
→Prob(X/Time(Return(Robert)) is C/≤∘*6:15 pm) is D/usually
Searching the Deduction Database, we find that the basic probability rule matches the protoforms of the data and the query
Prob(X is A) is B
Prob(X is C) is D
where
Instantiating A, B, C, and D, we obtain the answer to the query:
Probability that Robert is home at about 6:15 pm is D, where:
(b) The tall Swedes problem:
We start with the data
p: Most Swedes are tall.
Assume that the queries are:
q1: How many Swedes are not tall
q2: How many are short
q3: What is the average height of Swedes
In our earlier discussion of this example, we found that p translates into a generalized constraint on the count density function, h. Thus:
is most
Precisiations of q1, q2 and q3 may be expressed as
Considering q1, we note that
μnot.tall(u)=1−μtall(u).
Consequently
which may he rewritten as
q2→1-most
where 1-most plays the role of the antonym of most.
Considering q2, we have to compute
A:
given that
is most.
Applying the extension principle, we arrive at the desired answer to the query:
Likewise, for q3 we have as the answer
As an illustration of application of protoformal deduction to an instance of this example, consider:
p: Most Swedes are tall
q: How many Swedes are short?
We start with the protoforms of p and q (see earlier example):
Most Swedes are tall→1/n ΣCount(G[A is R]) is Q
?T Swedes are short→1/n ΣCount(G[A is 5]) is T,
where
G[A]=Σi Namei/Ai, i=1, . . . , n.
An applicable deduction rule in symbolic form is:
1/n Σ Count(G[A is R]) is Q
1/n ΣCount(G[A is S]) is T
The computational part of the rule is expressed as
1/n Σi μR(Ai) is Q
1/n ΣμS(Ai) is T
where
μT(v)=supAi, . . . ,AnμQ(Σi μR(Ai))
subject to
v=Σi μS(Ai).
What we see is that computation of the answer to the query, q, reduces to the solution of a variational problem, as it does in the earlier discussion of this example in which protoformal deduction was not employed.
The foregoing examples are merely elementary instances of reasoning through the use of generalized constraint propagation. What should be noted is that the chains of reasoning in these examples are very short, More generally, what is important to recognize is that shortness of chains of reasoning is an intrinsic characteristic of reasoning processes which take place in an environment of substantive imprecision and uncertainty. What this implies is that, in such environments, a conclusion arrived at the end of a long chain of reasoning is likely to be vacuous or of questionable validity.
Deduction (Extension) Principle:
Underlying almost all examples involving computation of an answer to a question, is a basic principle which may be referred to as the Deduction Principle. This principle is closely related to the extension principle of fuzzy logic.
Assume that we have a database, D, and database variables X1, . . . , Xn, with ui being a generic value of Xi, (i=1, . . . , n).
Suppose that q is a given question and that the answer to q, Ans(q), is a function of the
Ans(q)=g(u1, . . . , un), u=(u1, . . . , un).
I do not know the exact values of the ui. My information about the ui, I(u1, . . . , un), is a generalized constraint on the ui. The constraint is defined by its test-score function
ts(u)=f(u1, . . . , un).
At this point, the problem is that of constraint propagation from ts(u) to g(u). Employing the extension principle, we are led to the membership function of the answer to q. More specifically,
μAns(q)(v)=supu(ts(u))
subject to
v=g(u)
This, in brief, is the substance of the Deduction Principle. As a simple illustration, let us consider an example that was discussed earlier. Suppose that q: What is the average height of Swedes. Assume that D consists of information about the heights of a population of Swedes, Swede1, . . . , Sweden, with height of i-th Swede being hi, i=1, . . . , n. Thus, average height may be expressed as
Ave(h)=(1/n) (h1+ . . . +hn).
Now, I do not know the hi. What I am given is the datum d: Most Swedes are tall. This datum constrains the hi. The test-score of this constraint is
ts(h)=μmost((1/n) (Σμtall(hi))),
h=(h1, . . . , hn).
The generalized constraint on the induces a generalized constraint on Ave(h). Thus:
μAve(h)(v)=sup (μmost((1/i n) (Σiμtall(hi)))),
h=(h1, . . . , hn), subject to:
v=((1/n) (Σi hi)).
More Search Engine Examples:
Let's consider a search engine query in which a person age is desired. For example, the question is: “What is the age of Mary?” or “How old is Mary?” or “What is Mary's age?”
Templates:
This question can be scanned or parsed, to extract its components, as (for example) in the following shorthand notation or format: “Mary/Age?” The parsing is done using many templates for recognition of the pattern or grammar for a specific language (e.g., American English), dialect, topic (e.g., political topic), or method and type of speech (e.g., written, as opposed to spoken information or question). The templates are stored and designed by linguists or experts, in special databases beforehand, to be able to dissect the sentences into its components automatically later on, and extract the relevant and important words and information. The degree of matching to a specific template (e.g., for English grammar), to find (for example) the subject and the verb in the sentence, is done by fuzzy membership function and other fuzzy concepts described elsewhere in this disclosure.
One example for the template is that the symbol “?” at the end of an English sentence “usually” indicates a “question” type sentence. (The concept of “usually” (or similar concepts) is addressed elsewhere in this disclosure.)
For question-type sentences, one can have the following template (as a simple example) for the question “How old is Mary?”:
(how old?/verb (to be)/noun (person's name))
That simplifies to: (how old?/person's name)
Or, equivalently, one can get this template: (age?/person's name)
Or, equivalently, one can get this template: (Mary K. Jones/human/Age?)
For a regular sentence of “Mary is 40 years old.”, we will have the following template, as an example: (Noun (person's name)/verb (to be)/number/years/age)
Using the keywords or flag words (e.g., the usage of verb “is”), that simplifies to:
(person's age/number/years)
Or, equivalently, one can get this template: (Mary K. Jones/Age/40/years)
Or, equivalently, one can get this template: (Mary K. Jones/Age/40 years)
Obviously, many other choices of templates and grammar also work here, as long as there is consistency and brevity in the definitions and templates, to reduce the size and get the common features for batch processing, faster search, faster data extraction, better data presentation, and more efficient data storage. The good thing about templates is that it makes the translation between different human languages (or translation between speech and computer commands) much easier, as they tend to carry only pure necessary (bare bone) information, without extra words, in a predetermined order or format, for fast and efficient access, search, and comparison.
Removal of Ambiguities:
First of all, there is an ambiguity as which Mary we are talking about. If the prior conversation or context of the conversation makes it clear that we are talking about a specific Mary or person, e.g., “Mary Jones”, then the search does not have to get the age of all people with that name or nickname that it can find, and the search will be limited (in scope) to the age of Mary Jones, only. Of course, if there are more than one persons with the name of “Mary Jones”, one has to search for other identifiers or distinguishing parameters, such as her middle name, middle initial, age, social security number, address, father's name, husband's name, neighbor's name, friend's name, graduation date from high school, name of high school, nickname, pictures, tags on pictures, voice sample, fingerprint chart, other biometrics, or employee ID number, to remove the ambiguity, if possible.
Another information from context or background base knowledge is that Mary is a human, and not the name of a pet or doll, in which case the search would be diverted to another domain of age determination (e.g., for pets or dolls). Now, let's assume, for this example, that the context of the conversation or background knowledge (database) dictates or indicates that Mary is the name of a person, and furthermore, we are talking about Mary K. Jones, specifically. Thus, the question becomes: “May K. Jones/human/Age?”
In addition, for humans, one can distinguish male names from female names for majority of the names, stored in corresponding female and male (or human) name databases. Thus, we will have the following question: “Mary K. Jones/human/female/Age?” This is such a common question that we have a template in our template database for this type of questions: “human/female/Age?” or “human/Age?” Let's now consider the template “human/female/Age?” for this example. For our question template “human/female/Age?”, we will have relevant data and relevant questions, associated with such a template, designed or input previously by humans, community users, search engine company, or the computer (automatically, based on the prior results and training or learning from the past associations in similar situations or relationships), into the template relational database(s).
The relevancy and reliability of sources of information (or information itself) are discussed elsewhere in this invention disclosure, using fuzzy systems (and Z-numbers). So, we will not repeat those formulations here again.
Relevant Questions:
The examples of relevant questions are shown below. These are linked to the template “human/female/Age?”, by linguists, or machine/computers trained for this purpose, using neural networks and fuzzy logic system combination, forming relational databases, that grows in size by experience and time/training, manually, automatically, or both.
-
- “What is the age of the person's kid(s)?” or “What is the age of the person's oldest kid?” (Because, usually, one has kids within some age range. For female humans (in today's standard) (living in US), for non-adopted kids, mother's age is usually in the range of 18 to 45 years old, with a membership function that is not flat, more or less in trapezoidal shape. Thus, the oldest kid's age is a very relevant question or piece of information,)
- “What year did the person graduate from high school (or college)?” (Because people in US normally graduate from high school around the ages of 17-19, with a corresponding membership function.)
- “When did the person buy a house (or his or her first house)?” (Because a female person in US (or at different regions of US, or in certain city,or within a certain income bracket or job classification) buys her first house at a certain age, say, for example, around the ages 25-35, with a corresponding membership function.)
- “How old is the person's best friend?” (Because, “generally”, each person is about the same age as her/his best friend, with a corresponding membership function.) (Please note that the concept of “generally” (or similar concepts) is addressed elsewhere in this disclosure.)
- “How old is the person's pet?” (Because, usually, one's pet is younger than himself or herself, with a corresponding membership function.)
- “How old are the person's parents?” (Because, usually, one is younger than his or her parents by about 20 to 45 years, with a corresponding membership function.)
Combining all the questions above (and their answers or similar information), one can get a good estimate of the person's age, using fuzzy concepts shown in this disclosure. In addition, using a relevance scoring system, one can filter and find all or most relevant questions. Each relevant question can in turn refer to another relevant question or information, as a cascade and chain, bringing or suggesting more questions and information for the user. The history of the user or history of the users or history of similar or same question(s) can be stored in some relational databases with relevance scoring, for future filtering and usage, based on a threshold. The system is adaptive and dynamic, as well as having learning/training mode, because as the time passes, with more experience and history, the database gets more accurate and larger in size, to fit or find the questions or relevant information better and faster.
Similarly, for answers or information available, one can find relevant information, using a membership function for relevance degree. Some examples for answers or information are:
-
- “The age of Mary K. Jones's oldest child (or kid) is 15.”
- “Mary K. Jones graduated from high school in 1989.”
- “Mary K. Jones bought her first house in about 1996.”
- “Mary K. Jones's best friend is most likely 35 years old.”
- “Mary K. Jones's dog is roughly 10 years old,”
- “Mary K. Jones's mother is about 70 years old.”
Sometimes, one gets the age of Mary K. Jones indirectly, through the information about her best friend's parent's age, which typically has less relevance and less credibility, in the chain of connected information. However, in this disclosure, we have shown the tools to treat and analyze/process all of those situations and information, with different degrees of relevance and credibility, using fuzzy concepts, such as membership functions for corresponding parameters.
Note that to search information and questions, one can use the following templates for the following sentences, as examples:
-
- “Mary K. Jones's dog is roughly 10 years old.” is converted to the template: (person/person's pet/pet's age/roughly/10 years), which is stored in relational databases, which can be queried, compared, aggregated, edited, combined, re-named, indexed, or re-ordered.
- “How old is the person's pet?” is converted to the template: (person/person's pet/pet's age?), which is stored in relational database(s) or template storage.
Another example for the search engine is an inquiry about Mary's house: “How much is the price of Mary's house?” To analyze this question, a process and system similar to the one given above is followed. However, in this case, in addition, we have some predetermined templates for links to relevant web sites or government information repositories. For example, for price of the house, the average price of the houses (the trend) for US, city, region, county, and specific street or neighborhood become relevant, as well as, inflation, housing indices reported by Wall Street Journal or the US Government (e.g., the new permits issued for the last quarter or the current inventory of the new or old houses), and the size and details of Mary's house (such as the number of floors, number of garages, number of bedrooms, age of the house, and square feet of the land and living area), plus the recent real estate activities in the same area for similar size houses (from real estate repositories or county records for recent transactions). The prior sale prices of Mary's house, if any, with corresponding dates, are also relevant information.
Therefore, one needs some indices and data from newspapers, US Government, local government, county records, and real estate databases. These data are usually directly or indirectly available for search engines (assuming they are not protected by password or only available on subscription basis, which may need human intervention and input). The indirect ones may require proper question or another relevant data (or intermediary information) to link with the final answer. Thus, at the beginning, the people experts in economy and real estate are needed to design and set the links and relationships (or mathematics formulas and fuzzy rules or relationships between different parameters), as the initialization step. However, if similar concepts already exist in the rules and patterns or templates, the machines can initialize the new search links and populate the relationships, automatically, without any human intervention or input. The updates for the links or feedbacks can be done periodically by humans or users, or automatically by machines, e.g., by feedback from the history using a learning machine (e.g., using neural networks, trained to update the links or improve them, gradually, based on prior scores and past performances)
In the above example, the most important piece of information is probably the address of the house. A system for this example is shown in
So, after finding the address(es), the search engine is focused on any relevant information related to the found address, especially targeting the focused web sites and predetermined repositories that probably contain relevant and reliable information, as mentioned above. In case of multiple addresses, if we cannot resolve the real address among the multiple possible addresses (or if Mary may actually own multiple houses), we end up having a list of (multiple) possible addresses and their corresponding prices, with some certainty (or confidence) value or membership function, associated with each found address (and its corresponding price). The additional system components in this example are captured in
Another example for the search engine is an inquiry about the price of a car: “How much is the price of a low mileage 1991 Ford Mustang?” or “How much does a 1991 Ford Mustang (in a good condition) worth?” To analyze this question, a process and system similar to the one given above is followed. However, in this case, in addition, we have some predetermined templates for links to relevant web sites or commercial (or standard) information repositories, such as E-Bay web site, auction web sites, used car dealers, car advertisem*nt or newspapers' web sites, car collectors' web sites, car magazines' web sites, reliable car blogs, car experts' web sites, or Blue Book values for cars.
In addition, the mileage on the car, car condition, and details of the car are also relevant. In this case, we know that the car has a low mileage (or is in good condition), which is a fuzzy statement, with its corresponding membership values and function regarding mileage (and/or condition) of the car. The fuzzy analysis is discussed elsewhere in this disclosure. We do not know the exact details of the car, for example, the options or extra features on the car. Thus, we probably get a range of values for the car (to include various options or features)
Updating Information:
History and the results of the same or similar questions asked or searched earlier by others can be stored by the search engine company (or others) on different repositories for fast retrieval or updates. Some questions have answers which are time-dependent, such as the value of a dollar with respect to Euro's, which changes every day or every hour. Some answers do not change that much (or not at all). For example, the capital of France is Paris, and it probably does not change very often or very soon. Or, (2+2) is always 4 (in conventional mathematics). So, one can separate these questions into at least 7 categories (which is a fuzzy concept by itself, with assigned percentages being approximate fuzzy ranges of numbers, as well). It can also be defined as a crisp range. One example is:
-
- Things that never change. (about 0%)
- Things that rarely change. (about 1-10 percent)
- Things that seldom change. (about 10-25 percent)
- Things that sometimes change. (about 25-75 percent)
- Things that often change. (about 75-90 percent)
- Things that usually change. (about 90-99 percent)
- Things that always change. (about 100 percent)
The classification above is shown in system of
For example, in one embodiment, one can store the corresponding history and past answers in repositories which have different purposes, such as “long term repository” or “daily repository”. The “daily repository” is updated on a daily basis or very often. In addition, an unreliable otherwise “long term” answer (with low score or low membership value, in terms of reliability) will still be stored in a “daily repository”, because it should probably be changed or updated soon. Thus, fuzzy concepts determine where we put or access the prior results or history of prior searches. In addition, generally, all things being equal, a “daily repository” has a faster access or update time, because it is used more often by the search engine, as the short term repository or database.
In addition, as an off-line mode, one can do batch processing in advance on future anticipated searches that are common or possible, based on some “possibility” degree (which is fuzzy value by itself), to store the information in repositories for future fast access, without too much (or not at all) processing delay. The repositories are classified based on topics they carry information for (on a fuzzy set basis). See
Also, there are some dynamic assignment and updates as to where information is stored (or be restored), for faster access, because some topics or subjects may become very much searched for in a specific period of time or on a temporary basis (e.g., political candidates' names are generally searched very often just before the elections, and the search will go down drastically right after the election). The predictor engine (which predicts or stores such trends or patterns) and assign or engine or module (which assigns or re-assigns the storage location) periodically re-evaluate and re-assign the repository locations for various subjects and topics, to be more efficient, for search and access the data. The prediction, assignment, and topics themselves are all based on fuzzy concepts and fuzzy sets. See
Furthermore, some repositories are assigned as intermediary repository, as a hierarchical structure or tree configuration, to access certain data faster. Alternatively, the data can be split up and stored in pieces for faster search or access, in a distributed fashion, due to the size of the files or the purpose of the files. For example, title, text, video, and sound related to a movie can be split and stored separately, in separate databases, servers, or repositories, where just the titles are stored in a specific server for fast access and search (by title only). Then, after the title searches are complete (with low overhead) and a specific title is selected, the pieces or components of the movie can be retrieved from various locations. For sonic applications, this increases the efficiency of the search engine. The classification of purposes or tasks to assign various repositories (by itself) is a fuzzy concept, with fuzzy set(s) and membership function(s). (These were addressed elsewhere in this disclosure.) See
In one embodiment, to answer the question “What is the price of Mary's house?”, one tries to start from “Mary” and get to “her (Mary's) house price”. But, one does not know at the beginning that which subjects are relevant and how relevant they are. For example, is the price of her car relevant? Or, is the price of her dad's house relevant information? Or, is the address of her dad's house relevant information? What is the relevancy and to what degree? Is there any general rule or relationship connecting the 2 concepts? Is there any specific rule or relationship (just for Mary) connecting the 2 concepts? If so, what is the rule or relationship connecting the 2 concepts? Should we search for the other concepts and at what length or at what expense? Now, we address the above questions.
The computational expense is generally in terms of search time and computing expenses, e.g. using total CPU power by many servers or a server farm (e.g., using the unit FLOPS (or flops or flop/s) for floating-point operations per second, as a measure of a computer's performance), to justify or gauge how far we should search for a concept, as a fuzzy limit or threshold, to stop or limit the searches. Generally, the more relevant the subject (which is a fuzzy parameter by itself), the more computational expense or time is justified, allowed, or allocated for searching that subject or topic (i.e. the threshold for how long we can search for that subject is higher).
The relevance is generally not known at the beginning. So, the system guesses the best it can, and if during the search steps is proven otherwise, the relevance factor is re-adjusted accordingly (going up and down, based on the observations, performances, and satisfaction of the goals or scores, on the first search cycle). For example, the system may guess a few subjects that may be somewhat relevant to Mary's house price, but it is not sure about them. Based on the specific initial knowledge base from Mary and the general knowledge base from the Universe (all other available data), the system prioritizes those guesses and assigns some scores to those possible subjects (so that the relative or absolute computational times are determined and limited for those subjects or topics), using fuzzy rules for relevance scoring, described elsewhere in this disclosure.
Let's assume for this example that “the address of Mary's dad's house” is set as relevant (with a high degree of relevance, which is a fuzzy parameter). T hen, the system tries to step forward from both sides to reach each other. This approach is similar to digging a tunnel in a big mountain, from both sides of the mountain, but without the exact GPS information, trying to get to the other side, simultaneously digging and stepping forward from both sides, using the best guesses and knowledge available for the best direction for digging (which is the same as guessing the relevance of the next step or subject, and choosing the most relevant subject(s), in the chain of relevancy, as accurate as possible, with the current knowledge we have so far, to minimize the computational power needed to get to the result (to the other side of the tunnel)). For example, now, we have “the address of Mary's dad's house”, and from that, we want to get to “Mary's house price”. In the next step, if we assume that “Mary's house address” is relevant to the context of this problem, then we have the following situation:
We now have “Mary's house address”, and from that, we want to get to “the address of Mary's dad's house”. Now, we look at the rules in our universe of rules storage, and we find that there is a strong correlation (which is another fuzzy parameter) between the address of a person and her parents, in terms of street address proximity, neighborhood, city, or zip code. So, we now can connect the two sides. That is, we can connect “Mary's house address” with “the address of Mary's dad's house”. That is, from the address of her dad, we can choose the best address(es) for her house, from all possible choices so far, that “fits the best” with her dad's address (with a higher correlation factor). So, we can narrow down or pinpoint her address(es) (or choices of her addresses)
In addition, if we are dealing with 2 or more topics or subjects simultaneously, we can get to her address from 2 or more directions, adding more confidence to the final result (of her address). For example, using “her income” to get to “her address”. In addition to the above, we will probably get more confidence on her address, at the end.
The system described above is shown in
In one embodiment, to answer the question “How old is Mary?”, we are looking for relevant answers (or information, subjects, or topics) and relevant questions. If the relevant information is not readily obvious or available, we can generalize and expand the scope of the topics, to try to fish or search for new topics under the larger new scope. For example, here, we have: (Mary/age?), which can be generalized to a larger scope as: (human/age?), which (in turn) relates to (human/country of residence) & (human/gender) & (human/type of job). Therefore, we have increased our choices of relevant topics or subjects to: “country of residence”, “gender”, and “type of job”, which were not obvious at the beginning of the analysis. Thus, we can follow those leads, for topics for the search engine, to find the possible ages (or range of ages) for Mary. This is shown in
In one embodiment, one gets to the answer(s) by following multiple paths, starting from the question template, working toward the possible answer(s). In one embodiment, users can give feedback or score answers or paths traversed, for better future path selections. See
In one embodiment, the relationships stay the same, but the inputs may constantly change, resulting in a dynamic (constantly-changing) output. For example, Gross Domestic Product (GDP) of a country and the population of a country (the inputs) constantly change. So, GDP per capita (the output) also constantly changes, but the relationship between GDP, population of the country, and GDP per capita of the country (the relationship between inputs and output) never changes. Therefore, the relationships or parameters that remain constant are stored in different repositories (compared to those of the dynamic parameters), and are accessed without any updating or verification in the future. For example, the formula for GDP per capita is always the same, for the same country or other countries, and it does not have to be updated or reviewed again, making access to that parameter or relationship much faster and less costly for the search engine. The most common or most used parameters, relationships, definitions, or topics are stored in separate repositories, which are grouped and sub-grouped in different classes and categories according to their topics, in a tree-structure or hierarchical form, for faster and easier access by the search engine. In one embodiment, the grouping is done based on fuzzy definitions and sets/subsets, See
In one embodiment, the same information may have various representations with different levels of details: L1, L2, . . . LN, where L1<L2< . . . <LN, in term of “level of details”. So, we can store them in different repositories, available for different searches. Search and access to L1 is much faster than those of LN (which carries more details). Based on the application, if it is determined that there is no need for details of LN, one can choose a version with lower amount of details, such as L1 or L2. An example for this situation is when an image or picture is stored at different resolutions (with different sizes) at different repositories. Or, another example is when a table (or spreadsheet or database) is stored, with various sections or columns are hidden or disabled (and not stored), so that different versions of the table (with different sizes and details) are stored in different locations or repositories, and each version of the table may fit or serve different types of user, application, need, search, or query. The level of details can be expressed (by the user) as a fuzzy parameter, for the original file or data. See
In one embodiment, there are 2 types of information (static and dynamic) which yield the same result(s). For example, for Mary's age, one can store the information as “39 years old” (dynamic information, which changes every year). Or alternatively, one can store that same information as her exact birth date, as an equivalent data, which is always static (not changing). The second method or type (static information) is more useful for the future referrals. For example, once the today's date is known, the birth date is always useful (and complete information) to calculate the age of a person, whereas the age number or value (from an unknown number of years ago) (by itself) is much less useful (and less complete, to calculate the age of the person). Thus, one can store the static information separate from the dynamic information, as they are accessed differently, with different priorities, access frequencies, and degree of “usefulness” (which can be expressed by fuzzy concepts), to optimize the search engine, especially for future searches on similar topics. See
Familiar or Famous Names or Titles:
In one embodiment, famous names and titles are stored and indexed or ranked separately, for fast and efficient access, e.g., Eiffel Tower, Clinton (referring to the former US President Clinton), Paris (referring to Paris, France), or The US President. There are 2 types of famous names and titles. The first type has a single choice only, with no ambiguity (e.g., Eiffel Tower), but the second type has more than 1 choices, with some degree of ambiguity (or membership value). For example, we have more than one city in the world called Paris, and Paris is also the name of a person, as well as the name of a Las Vegas hotel and casino. However, “Paris” by itself (without any context) most likely means “Paris, the capital city in France”, as our first choice. Other choices can be ranked as a list (with some membership value), but the ranking can be changed based on the context, e.g., prior sentences, history, background, speaker, audience, or the location of the conversation. In addition, in one embodiment, the 1st and 2nd types are separately stored and listed, to streamline the process, for more efficient search engine access. See
In one embodiment, some titles are placeholders, e.g., The President of the United States, which is expected to have possibly different values every few years, which should be checked and updated, according to that time periodicity, e.g., every 4 years, starting from an election year in US. This means that some repositories are tagged and treated that way, for optimum performance, e.g. more accuracy and less frequency of updating of the data (or less required computing power or expense). See
In one embodiment, there are the accuracy factor and reliability factor involved in the search engine, in addition to the cost factor for computing power (used so far, for the search engine). That is, there is a threshold as to how much accuracy we need for the result (which could be a fuzzy parameter itself). As an example, we may need to find (and search for) the diameter of the planet Earth to 10 km accuracy (rather than 100 km accuracy). Thus, we generally have to do more search to get that much confidence or accuracy (with enough reliability) (i.e. for 10 km accuracy (rather than 100 km accuracy)). Another example is to find the value of real number “e” to 5 decimal point accuracy (rather than, for example, 2 decimal point accuracy). There is also a threshold as to how much computing time or money we want to spend on this search, which means that how bad we want the answer, and how long we are willing to (or allowed to) spend on this search. Thus, accuracy, reliability, confidence, and cost are some of the factors that determine the scope and depth of each search. All of these factors can be expressed as the fuzzy concepts, as explained elsewhere in this disclosure. See
In one embodiment, storing the prior results or calculations (or intermediate results), especially when they are requested multiple times or very often by other users or the same user, increases the efficiency of searching same or similar terms or topics in the future, similar to the way humans gain experience, learn, and store information, for future recollection. The storage and recollection of the prior information is done in multiple steps. First, the information is scanned or parsed (e.g., a birthday event for a person) for its parameters and characteristics (e.g., cake shape, cake taste, birthday song, colorful hat, friends present, and gifts received). Then, it is tagged or indexed based on those parameters and characteristics (e.g., song, cake, taste, shape, hat, gift, friend, human, and food). Then, it is stored based on the tags or indexes in proper repositories. There are multiple classes of repositories, e.g., in terms of short-term and long-term, e.g., for frequency of access or access speed for retrieval (or access speed for editing and updating information already stored). So, there is a processor or controller which makes that classification (which can be fuzzy, as well), for proper storage. See
Then, there is an association tag or pointer that points the subject to another similar subject (e.g., with a similarity degree, being expressed as a fuzzy concept, as well). For example, the taste of the cake (being a chocolate cake) is a reminder of the other subjects or topics, e.g., “chocolate” or “hot cocoa”. Thus, it would point to “chocolate” or “hot cocoa”, or both, with a pointer(s). In one embodiment, the association pointers can point to other subject pointers, as N cascaded or chain of pointers in series (or combination of series and parallel configurations), where N is an integer bigger or equal to one. In one embodiment, the links in the chain have different (non-uniform) strength, indicating the different degrees of associations between pair of chained subjects. In one embodiment, the association is among M subjects, where M is bigger than 2, e.g., 3 subjects, which are all related to each other. See
In one embodiment, the association can be with an event, such as “cake dropping on the curtain”. Thus, it points to the subject “curtain” or “stain” (which in turn points to “circular marking” and “circle”). One way for recollection is to store the links or end of the links (or pointers or pointed subjects), and follow the chain or link backward or forward to get the result from either sides, or even start from the middle of the chain and continue in one direction, to recover or find the original subject. So, each subject can trigger another one through the chain sequence. See
In one embodiment, for long term storage, one puts the information or chain of associations as a whole (or broken into some components or parts, or even sampled e.g., every other subject in the chain, to store less data, as a lossy storage, to save storage space) into long term repositories (for not-frequent access or not-in-near-future access). Note that for the recollection of the broken data or lossy storages, one requires some computing power to reconstruct the lost links later on (by associating pointers), to reassemble the jigsaw puzzle, as the original chain. See
In one embodiment, when parsing sentences using our methods described here in this disclosure, to search for a more accurate meaning, among possible meanings, especially in a specific context, we can narrow down the choices or targets, as a whole sentence, because the possibility of adjacent two or more words to have a coherent meaning or consistent interpretation eliminates most of the initial individual possibilities for a single word, when presented as a sequence of words in a specific order (or chain of pointers between words).
Note that a human brain carries information and memories as encoded patterns of neural firings.
In one embodiment, the system (of our invention) stores the information for our search engine in the distributed memory repositories. In one embodiment, the links or pointers between subjects get deleted, by the system, if the pointers or links are not used for a long time, to recycle the released memory, as available, for future use. For example, periodically, the system checks for unused links that are idle for a long time (a fuzzy variable), to release the memory location (and break the link or pointer), if applicable.
In one embodiment, the links or pointers between subjects have various weights. That is, the links are not uniform in strength. Or, the link between two subjects is not binary (e.g., “linked” or “not-linked”). For example, the link strength can be expressed as a real number between 0 and 1. The higher the value of the link strength, the more correlation exists (or more correspondence) between the two subjects. Variable strength link between two subjects can also be expressed in the fuzzy domain, e.g., as: very strong link, strong link, medium link, and weak link, as shown in
In one embodiment, social network sites provide feedback of the users and connectivity between users as an indication of the trend or pattern of society, groups, or individuals, with respect to different subjects, such as taste in music, marketing directions, or political opinions. Thus, they are good databases for data mining. Tweeted subjects (on Tweeter feed traffic monitoring module) can also be studied and classified to find patterns and extract data, for marketing and political purposes, e.g., as to who may become the next president of the United States, e.g., by counting or getting the frequency of a name or subject at a specific time. See
In one embodiment, one can use the search engine to predict the price of airline ticket for next vacation for next month, or find the current best price or option available (or best travel plan), considering the travel constraints or rules that we impose. In one embodiment, the search engine can also be used to find the best route to drive home from airport, considering the rules and conditions, with traffic constraints or how much gas we have, to minimize the driving time (as an example). In one embodiment, the price of a company's stock or average group of stocks is predicted for next month, or the best stock value is distinguished, among many companies, based on the rules and constraints about their products and the industry, using fuzzy analysis, explained elsewhere in this disclosure. See
In one embodiment, the search engine displays the source of the information for the user, e.g. “Wall Street Journal”, as the added value for the search result, which accompanies the credibility of the source, e.g., as a fuzzy parameter. In one embodiment, the search engine focuses on web sites to return personalized results, based on previous browsing habits of the user. In one embodiment, the user inputs personal information to customize the search results or help the search engine go to the right or more relevant direction, with respect to the user's preferences, taste, or parameters. For example, knowing that the user lives in San Francisco or vicinity area (in California, USA) (as her resident address, as one input by the user, through the user interface module), the search for “football team” yields “The San Francisco 49 ers” (which is a professional American football team based in San Francisco, Calif.), and this result has a higher ranking or score than another American football team in another city, and this result also has a higher ranking or score than a soccer team in San Francisco, Calif. (because “football” (generally, in US) refers to the “American football”, not “soccer”). This means that the meanings of the words are clarified and set based on the context and background information, e.g., user's information or preferences, such as address, zip code, ethnicity, religious, weight, height, age, gender, job, income, political affiliations, college degree, food preferences, health information, marriage status, type of car, or the like. See
Similarly, in one embodiment, the prior queries help customize the search result for future queries. Other factors can be how many times or how often a user (for example) searches for food or nutritional facts, and how long the users spend on a web site related to the food. This interest in food-related subjects makes “food” a more relevant subject for that user for future, to be a factor for relevance determination of other subjects in the search. In one embodiment, the user allows that the search engine tracks her usage and habits or patterns, from the user-input module, e.g., through the menu on screen, for privacy level settings, which can also be another fuzzy parameter. See
In one embodiment, the search engine tracks the music, books, movies, and videos that the user downloads, buys, rents, listens, watches, or looks at. In one embodiment, the search engine tracks the user's emails and the patterns related to the emails or SMS, e.g., the recipients, how often sent, what time of day sent or received, any attachments to the email, what type of attachments to the email (type of file, e.g., JPEG or PDF), size of the file of the attachment or the email, or the like. All of the above parameters indicating the degrees or quality can also be expressed as fuzzy parameters. In one embodiment, the search engine has a user-interface or GUI (graphical user interface) for the user inputs, with scaling or sliding bars, knobs, or selectors. See
In one embodiment, the search engine or a software agent/bot goes into email list or friends list, and find who may be, e.g., Chinese, from the possible cultural signatures or names or last names or other tags or info about that person, to guess the origin of that person, to gather all of those people automatically under one group name (which alternatively can be tagged manually), where it can use the compiled list to invite all of those friends automatically for the Chinese New Year party, by calendar date trigger automatically, one week before the event, as an example, or send an email to all of those to congratulate/send good wishes for the Chinese New Year, or do a voice mail or do a jingle or slogan or music or poem or short message, by sound, text, video, multimedia, video, image, or the like.
In one embodiment, the search engine connects to the modules for controlling ads, coupons, discounts, gifts, or filters for web sites (e.g., filters deleting specific web sites for children, from the search results). In one embodiment, the search engine rewards the user on points for discounts for purchases or coupons, in exchange for giving up some privacy, for personal information input by the user. In one embodiment, the search engine is self-customized engine or module that can be embedded on a web site. In one embodiment, the search engine helps the ads targeting a user, based on personal information, such as birth date, e.g., for gift suggestions, or statistics or biometric-driven, such as user's height or user's household's income percentage, with respect to those of national average or median. See
In one embodiment, the user specifies her purpose of the search, e.g., medical, business, personal, or the like. For example, searching for a hotel yields different results for a business trip (near convention center or downtown), versus for a vacation trip (near the beach or amusem*nt park). In addition, for example, specifying the accompanying persons can modify the search results. For example, having kids with the user during a vacation trip tilts or focuses the search results toward the vacations, hotels, or cruises that are tailored to families and kids (family-friendly or oriented), whose information can be extracted from the tags or scores supplied by the hotel itself or its web site, e.g., meta-tags or metadata, or from the tags or scores supplied by other users, or from the text comments or feedback by other users about their experiences with that hotel. See
In one embodiment, the user asks a question, and the search engine first determines the language of the question (e.g., by parsing the sentence or question), or the user herself supplies the information about the language, e.g., French. The search can be focused on web sites in French language (e.g., using the metadata or flags from the web site), or search any web site, depending on the user's or default settings for the search engine. In one embodiment, the search is on one or more of the following formats (and the search results are also in one or more of the following formats) text, web sites, links, emails, video, images, line drawings, paintings, satellite images, camera images, pictures, human pictures, music, blogs, HTML, PDF, sound, multimedia, movies, databases, spread sheets, structured data, slides, or the like (or a combination of the above), per user's setting or default. See
In one embodiment, the search engine is queryless, i.e. with no questions at all, but the search engine provides or suggests some subjects or topics automatically, sua sponte, based on the history and user's preferences or prior user's feedback. In one embodiment, the tagging, scoring, and feedback can also come from friends, social network, other users, similar users, club members, or co-workers, e.g., using bookmarks, links, and shared searches, presented, displayed, or forwarded to others. In one embodiment, there is a biometrics or security module associated with the circle of friends or social network, to protect the shared information, against unauthorized or free access or hacking. See
In one embodiment, the search engine and corresponding natural language parsing and processing are tailored toward the specific application or industry, e.g., telecommunication, stock trading, economy, medical diagnosis, IP (intellectual property), patent, or claim analysis or valuation, company valuation, medical knowledge, and the like. For example, a lot of abbreviations and words have very specific meanings in a specific technology, context, or industry, which may be very different in other contexts or environments, causing uncertainty or misleading search results or language construction or interpretations. For example, “IP” means “Internet protocol” in telecom industry, but it means “intellectual property” in patent-related businesses. To minimize those negative effects, the user specifies the industry from the beginning. The modules can be trained for various industries, and they can be separately sold or accessed as a service for specific industry. See
In one embodiment, using common rules for grammar and syntax for a specific language for sentence structure (and corresponding exceptions to those rules), the search engine parses and dissects the sentence (as explained elsewhere in this disclosure) and applies dictionaries (in different categories, such as medical dictionaries) and thesaurus (or phrase books or glossaries or idiom or phrase or dialect listings) to find or interpret the meaning of the words, phrases, and sentences, e.g. to convert them into codes, templates, abbreviations, machine codes, instructions, text, printout, voice, sound, translation, script, or computer commands, to process further, if needed. See
In one embodiment, the synonyms module, spell check module, antonyms module, and variation or equivalent word module are all part of a search engine, to help find similar words and concepts, or parse the sentences. In one embodiment, for analytics, the search engine includes summarization module and clustering module, to group the data in sets for systematic analysis, such as based on N-dimensional feature space for components of a word or phrase, based on all the possibilities for basic components, partial words, or letters in a given language (as a dictionary for all possible basic word components in a given language, with all connecting possibilities with other neighboring components, which is held in a database(s) or relational databases, and can be updated and improved by users periodically as feedback, or by machine or processor, automatically, with a training module, such as a neural network).
In one embodiment, social bookmarking, tagging, page ranks, number of visitors per month, number of unique visitors per month, number of repeat visitors per month, number of new visitors per month, frequency and length of visits for a given web site or web page, number of “likes” or “dislikes” feedback for a site or topic from users, and number of links actually requested or existing for a web site, as absolute or relative numbers, or as a rate of change (first derivative) of the parameter, are all parts of the search engine analytics, for finding the more relevant search results, with respect to a specific user or general public users. In one embodiment, tagging and user comments are done as an annotation to search results, as an extra layer. In one embodiment, what other people, users, or friends have done is displayed or suggested to the user, e.g., actions performed or web sites visited or items purchased.
In one embodiment, a search is personalized or customized using the position or role of a person in an organization, e.g., CEO of a company, with her parameters pre-set as a generic CEO, and can be further defined based on specific personality of the CEO, by herself, in such a way that a new CEO does not have to change the pre-set generic or basic part of the profile, making the transitions much smoother for a new CEO. The role-based model can be combined with the concept of inherency, so that a class of roles or positions can be defined categorically (only once, in a very efficient way), and then, subclasses may have extra features, conditions, or constraints on top of those of the corresponding class.
In one embodiment, live search is conducted using human experts as helpers, to guide the searches in a general direction by input phrases or feedbacks, in a limited scope, interactively with machine or computer. This is useful for a new field, in which not much information is accumulated in the databases, and most of the information is in the head of the human experts at this early stage. In addition, the user base and number of queries are manageable (small enough) with a few experts on line. This is not scalable or cost effective for large user base or databases, with too many queries to handle by human interventions.
Pattern Recognition:
In one embodiment, the images are searched for specific color or patterns or shapes, e.g., for houses or clothing, to match a target or find one similar to a target, based on the features defined in feature space, such as stripes patterns, color red, circles, dot patterns, trapezoid shape, or the like, as a pattern recognition module, looking for degree of similarity, e.g., as a fuzzy parameter, for real estate agents to search databases and sell houses or for department stores or store web sites to sell clothing to potential customers. This is also useful for analyzing and classifying Facebook® and photo album sites, e.g., for face or iris recognition, e.g., to identify, track, or classify people or objects. This is also useful for the security purposes on Internet or by cameras at the airports or buildings.
In one embodiment, the video is searched, using still images, motion vectors, and difference frames, e.g., to find a car or face in the video, to find the speed of the car from the location of the car in different frames, or to recognize a person in the video, using face, iris, or eye recognition (or other biometrics), or target tracking objects in video frames to get other identification parameters or features from the video. This is also useful for analyzing and classifying YouTube or movie repositories or music videos, e.g., to find or track people, subjects, objects, topics, or songs.
In one embodiment, the video track and sound track from a movie can be separately analyzed, for various sound and video recognitions, such as spotting some sound signatures or sequence of notes, indicating an event or music, or using voice or speaker recognition (as explained elsewhere in this disclosure), to find or recognize a person and tag or classify the track or movie. In one embodiment, the recognition engines or search engines from different tracks are combined or compared with each other, to get a better result, with more confidence, faster.
In one embodiment, the maps or road maps are scanned and analyzed to get (for example) geographical or residential information, for civilian or military purposes, e.g., for market search or business intelligence gathering. Markings, captions, scales, symbols, and names on the maps are recognized by OCR or pattern recognition module, to interpret the maps and find people and locations of interest. For satellite images, the objects have to be recognized, first (by object or pattern recognition module), as what they are, and then they can be categorized or classified (by tags or flags), with comments, text, or identifiers superimposed or attached to the image file. Object recognition with possibility of choices is expressed in fuzzy system, with membership values, e.g. recognizing an object as a bus or truck in a satellite image.
In one embodiment, Wikipedia and other encyclopedia or informational sites are referred to by the search engine for search on the topics they carry. In one embodiment, the search engine categorizes as how often a web site should be reviewed or searched based on how often it gets updated (in average), how relevant is the web site for our topic of search, and how reliable is the source of the web site. For example, the more often it gets updated and the more relevant and reliable the web site, the more often the search engine would check the web site for updates and new information, to search and extract data. In one embodiment, the search engine tracks and analyzes the web site traffic, for patterns and information about the web site, including for the web site reliability analysis.
In one embodiment, all the units of weight, length, and the like, with the corresponding conversion factors are stored in a database, for example, to convert “meter” to “foot”, for unit of length. The physical constants and physical, chemical, or mathematical formulas or facts (e.g., as relationships or numbers), such as speed of light or formula for velocity in terms of distance and time, are also stored in corresponding databases or tables, for easy and fast access for the search engine, e.g., with a hierarchical indexing structure or relational database(s). Alternatively, the search engine can refer to reliable web sites with similar information, for search and extraction of data.
In one embodiment, the components (such as text, video, and sound track in a movie data) are separated and searched separately, on an optimized and dedicated search engine for that format of data. See
In one embodiment, the system uses the tags or comments written by various users, or searches and parses those comments to dissect or convert them to the individual tags. (The example or method of parsing of a sentence or phrase is given in another part of the current disclosure.) This way, the collection of knowledge or intelligence of many users and people are combined to find a better or faster match(es) for the search. One example is the pictures tagged by the users, which are searchable in different databases, to find a correspondences or likelihood of relationship between one name and multiple pictures.
On the first cycle, the fuzzy classifier module or device classifies or separates different pictures into clusters or groups in N-dimensional feature space. For example, it uses facial features and parameters or biometrics, e.g., the approximate length of the nose, or ratio of width of the nose to length of the nose (as a dimensionless number or normalized parameter or other features related to iris or eye recognition. This corresponds to multiple individuals having the same exact or similar name. Please note that “similar name” is a fuzzy concept, by itself, with its own membership function value.
On the second cycle, it further distinguishes between or finds pictures of the same person in different ages or in different forms or settings (such as with dark eyeglasses, or wearing fake or real beard or mustache, or wearing scarf), which as the first filtering pass or cycle, it may look or get classified as a different person. One way to find the right person is the use of biometrics parameters, such as eye and nose, that “usually” do not change by age “that much” for the same person. Please note that “usually” and “that much” are also fuzzy parameters and concepts, by themselves. The other way is the correspondence of the date that the picture was tagged or posted, which may correspond to the date of the original picture, or equivalently, to the age of the person in the picture (or the year the picture was originally taken or captured). The other way is the comments or text or tags by the users that accompany the pictures, which collectively give probability or correlation for the identification of such person. The other way is the correspondence of the item of clothing (or attached objects, external items, context, environment, or surrounding), e.g., wearing the same or “similar” shirt or neck tie in 2 different pictures. Note that “similar” is another fuzzy parameter here.
Even, more general is the correspondence of the preferences or characteristics of the person, as a collection or set of parameters. For example, for a person living near the beach in Florida (e.g., a Miami Beach address as residential address), the system expects higher probability of casual dressing, bathing suit, sun glasses, and tropical trees appearing in the picture. So, those features appearing in a picture (e.g., casual dressing, bathing suit, sun glasses, and tropical trees) favors or increases the probability of a person with Miami zip code or address (or a person on vacation near beach), for identification purposes of a person in a picture, instead of a person with an Alaska address (or a person with no travel habits or history in tropical or beach areas).
Another example is that if a lady has many pictures with a red dress (or striped T-shirt or particular hat or design or designer or signature or pattern or style or brand or trademark or logo or symbol, e.g., a Polo shirt with its logo on it), the system can assume that the person has a lot of red dresses or prefer the color red for dress or shoes or car. Or, the red color preference is obtained from the user herself or her friends' input, as preference or history files (or based on a detective work file, by a third party, or by a software agent searching all over Internet for a person's personal data, or by marketing databases from a Macy's department store, based on past behavior or purchases, as her file history). Thus, if a person is sitting in a red car or wearing red shoes, in a picture or a video, it has a higher probability to be the person in question, based on her past or characteristic files, for identification or recognition purposes, e.g., for searching through Internet or databases to find all pictures or videos related to a name or a person.
The recognition of a pattern, color, person, face, logo, and text, including OCR (optical character recognition), is generally done by dissecting the image or video into pieces and components (including motion vectors for video, to track the objects, between the frames, as the difference between the neighboring frames) to find features or objects, and from the parameters associated with those features and objects, e.g., geometrical lengths or ratios or angles, the system finds or guesses the identity of those features or objects, based on sonic certainty factor or membership value (which is a fuzzy parameter). For an object with images captured from multiple angles, the data can be more useful, as it gives the information on 3-D (dimensional) object or depth, for better recognition.
For a pattern recognition module, we have an image analyzing system, e.g., as shown in
In one embodiment, a decision or discriminant function (an N-dimensional pattern vector) is used, to find the pattern class memberships and the fuzzy decision boundaries between different classes. For matching, in one embodiment, the system uses a minimum distance classifier, with each pattern class being represented by a prototype or mean vector, P:
P=(1/N) ΣXi
where N is the number of pattern vectors, and X is a pattern vector. Then, the Euclidean distance to determine the closeness is determined as, D:
D=∥Xi−P∥
where
∥K∥=(KTK)0.5 (It is the Euclidean Norm.)
The matching can be done by correlation, C, as well, between A and B, in another embodiment:
C(x, y)=Σg Σh A(g, h) B(g-x, h-y)
The correlation function may be normalized for amplitude, using correlation coefficient (e.g. for changes in size or rotation).
In one embodiment, an optimum statistical classifier is used. In one embodiment, a Bayes classifier is used, to minimize the total average loss (due to incorrect decisions), e.g., for the ones used for Gaussian pattern classes. In one embodiment, a perception for 2-pattern classes is used. In one embodiment, the least mean square (LMS) delta rule for training perceptions is used, to minimize the error between the actual response and the desired response (for the training purposes)
In one embodiment, a multi-layer feed-forward neural network is used. In one embodiment, the training is done by back propagation, using the total squared error between the actual responses and desired responses for the nodes in the output layer. In one embodiment, the decision surfaces consisting of intersecting hyperplanes are implemented using a 3-layer network.
In one embodiment, for pattern recognition, the system uses the structural methods, to find the structural and geometrical relationship for a pattern shape, using a degree of similarity, which is associated with a membership value, which is a fuzzy parameter. In one embodiment, a shape number is defined for the degree of similarity. In one embodiment, a four-directional chain code is used to describe the shape. The distance between 2 shapes is expressed as the inverse of their degree of similarity. So, for the identical shapes, the distance between the shapes is zero, and their degree of similarity is infinite. In one embodiment, for shapes, the system uses similarity tree and similarity matrix to evaluate the degree of similarity, which can be expressed as a membership function, which is a fuzzy parameter.
In one embodiment, for shapes, the region boundaries is coded as strings, with the number of symbols matching as an indication of the degree of similarity. In one embodiment, for shapes, polygonal approximations are used to define different object classes. In one embodiment, a syntactic method is used to recognize the patterns. The system uses a set of pattern primitives, a set of rules (grammar) for their interconnections, and a recognizer with the structure defined by the grammar. The regions and objects are expressed based on strings, using primitive elements. The grammar is a set of rules of syntax, which governs the generation of sentences from the symbols of the alphabets. The set of sentences produces a language, which represents pattern classes.
In one embodiment, we represent the string grammar as a 4-tuple, (A, B, C, D), for the strings, with e.g., A, B, C, and D representing non-terminals (a set of variables), terminals (a set of constants), the starting symbol, and a set of rules, respectively. Then, objects or shapes can be expressed mathematically, by first conversion into its skeleton (using image processing on pixel level, for example, to thin down the image to get the line structure shape), followed by primitive representation (for example, basic structure or geometrical shapes, from database, to replace the skeleton), followed by structure generated by regular string grammar (to resemble the original shape, region, or figure). String recognizers can be represented using nodes and arrow connectors between the nodes in a graphical manner, similar to a state diagram.
In one embodiment, the string grammar can be extended or generalized into the tree grammar, for syntactic recognition of the trees, using a 5-tuple, (A, B, C, D, E), with E representing a ranking function to represent the number of direct descendants of a node with a label which is terminal in the grammar. Again, objects or shapes can be expressed mathematically, by first conversion into its skeleton (using image processing on pixel level, for example, to thin down the image to get the line structure shape), followed by primitive representation, using a tree grammar, to resemble the original shape, region, or figure. Selection of the primitives in this case is based on the membership values, and thus, it is a fuzzy parameter.
For recognition, the system deals with various knowledge base databases: procedural knowledge (e.g., for selection of parameters and procedures), visual knowledge (e.g., angle of illumination, producing shadow and other visual effects), and world knowledge (for relationships between objects, e.g., in an image of a car, the system expects to find or detect one or more tires under the car, if it is visible in that perspective), which sets the expectation in an image for higher validation, consistency, and accuracy. For example, for the world knowledge, the fact that “Cars usually have 4 tires.” can be expressed as follows:
[OWNERSHIP (car, tire, 4), USUALLY]
Or, it can be rewritten as:
OWNERSHIP (car, tire, at least 1)
Or, it can be expressed as: (“For all” cars, “there exists” one tire):
OWNERSHIP (∀ car, ∃tire)
These statements can be combined with others using logical or relationship operators, e.g. AND, OR, NOT, XOR, and IF-THEN statement (rules). Using the rules and relations, the system performs inference or deduction, using an inference module or deduction engine or device. The term USUALLY adds the Z-number to the statement of the world knowledge. Thus, if the system detects an oval or circular object in the image under the body of the car structure image object, then that may be a tire of the car. The tire detection can be expressed based on membership values, which is a fuzzy parameter.
In one embodiment, semantic networks are used, with nodes representing objects and the arrows representing the relationships between the objects. For example, for the example given above regarding “a car having a tire”, one node is CAR, and the second node is TIRE, with an arrow connecting the node CAR to the node TIRE, representing OWNERSHIP relationship between the 2 nodes.
Another example is the application of the position of two objects with respect to each other. For example, for a statement of “a car located above a tire”, one node is CAR, and the second node is TIRE, with an arrow connecting the node CAR to the node TIRE, representing ABOVE (positional) relationship between the 2 nodes, representing the 2 objects CAR and TIRE. The knowledge of the possibility of the existence and position of a tire in the image of a car helps the identification of the objects in the image (more accurately and faster). In addition, if the system is given another fact or statement that “A tire has a star-shaped rim,”, then if a star-shaped object is detected in the middle of the object of TIRE in the car image, then that star-shaped object may be the rim for the tire of the car.
The above example also works for facial features, e.g., for iris, face, or identity of a person recognition, in which there is a relationship between relative size and position of different components of eye or face of a human. The above example also works for spelling or word recognition (e.g., OCR) and voice recognition, in which there is a relationship between different sounds or letters that make up a word or sentence, for a given grammar and language, e.g., American English, in terms of sequence of the letters that make up a word or phrase or written sentence, or sequence of sound bites or tones or notes or frequencies that make up a speech or voice or spoken sentence. So, for all of the above, the relationship or relative position of one object or feature with respect to another is known, which helps the detection and recognition (or verification and confirmation) of all features and patterns in the image or in any other media.
In one example, if the comment or tag for a picture refers to “The last 4th of July with Clinton in the office”. After dissecting, parsing, and analyzing the statement (as described elsewhere in this disclosure), for a user in the United States of America (context-specific for the user), the phrases “4th of July” and “Clinton in the office” is probably a reference to “the former President Bill Clinton, of the United States of America” (based on the correlation of the words or concepts, or combination of the words, or order of the words in a phrase). The last 4th of July of President Bill Clinton's presidency (from the historical facts and databases, available to the search engine) is Jul. 4, 2000, Thus, the picture is tagged by a statement which refers to the date of Jul. 4, 2000. Having a date associated with a picture or piece of data usually helps to find the owner of the picture or identity of the objects in the picture or things associated with the picture (based on correlation, association, or probability), e.g., the identity of the person(s) in the picture. Note that the dates associated with a picture may generally be multi-valued, fuzzy, a range, or approximation date(s).
Note that in the example above, “Clinton” (extracted from the sentence and distinguished as a possible given name or family name) is already stored in a database for the famous names or people, with the following ranking order: (1) President Bill Clinton (as the more probable candidate); (2) Secretary of State Hillary Clinton; and so on. If there is no other supporting information available, the system tries the choices from the highest to the lowest. For the first choice (President Clinton), the “office” refers to the “White House” or “presidency”. In fact, the generic or common words in a language may have a specific meaning or different meaning, once it gets associated with another word, concept, context, or environment (e.g., politics, versus medical field). Thus, once a context is set or determined (such as politics or politicians), the specific database(s) for that specific context is activated or referred to, instead of the general or generic databases, to find the more exact or better meaning of the words or phrases. This is shown in
In an example, one name is very similar to another name in spelling or sound. Thus, during typing or conversion from sound to the text, the spelling may come out differently. In addition, names in different scripts such as Arabic, Persian, or Chinese may end up differently during conversion to the English or Latin script or alphabets. This uncertainty of the sound or spelling is captured in a database for a variation of a name or word, as possible candidates with different membership values, which is a fuzzy parameter. The database can be filled up and corrected by the users in a community of users. Another way is to have candidates for a partial word or sound, e.g., as the most common mistakes or errors, e.g., to find the final word with the correlation analysis, e.g. based on the scoring the combinations, and maximizing the score of the combination for all candidates. In an example, the partial word candidates are stored separately.
One example of the common mistakes is from the proximity of the letters on the typical keyboard, e.g., Qwerty keyboard, e.g., with R and T in the close proximity, making it likely for a person to type R, instead of T, e.g., typing RANK, instead of TANK (or typing TTANK, instead of TANK). In the cases that the mistaken word has a meaning, the mistake cannot be found by the spell check alone, and it can only be found through context analysis, e.g., for the phrase “water tank on the roof”, it would be understood by the system that the phrase “water rank on the roof” is just a typo or misspell, because the second phrase does not have a proper meaning.
Once the flag is raised about the improper meaning or misspell in the recognition system, one of the tests that the system does is to try and test similar words or phrases with similar sound or spelling, e.g., testing neighboring keys on the keyboard for possible mistakes, by replacing them in the suspected word, to see if any of the results has a proper meaning. Then, the system ranks the results, and it marks the result that has the highest score in the context of the phrase or sentence, for possible candidate for the original (correct) word.
To analyze a phrase or sentence, in one embodiment, the system looks at adjectives or related words, e.g., “water tank”. For example, for “tank”, when used as a word equivalent to a “container” (which can be extracted from the context, from neighboring words or paragraphs), it logically can hold some objects, especially fluids, e.g., gas, liquid, water, nitrogen, and liquid nitrogen. Thus, one can combine them this way, as a template:
[FLUID+tank]
Or:
[tank of+FLUID]
One can store these templates (and any exception to the templates) in multiple databases, which can be categorized and separated based on their topics and usages, in a hierarchical or tree or pyramid structure, with inherency property, e.g., parent nodes and children nodes.
This can be done with adjectives, as well, for example, “big” in the phrase “big tank”, which is expressed as a template:
[ADJECTIVE+tank]
Now, when we are scanning the sentences or phrases, we are using (searching for) the stored or pre-recorded templates in databases or storages, to find the patterns mandated by a template. Once a template is found (to match the pattern of a given sentence or phrase), the system can understand the meaning of that section of the text, phrase, or sentence. Then, it can understand the meaning of the whole sentence or phrase through the combinations or series of templates that construct those phrases and sentences (for a given language, based on the collection of the grammar templates (along with their exceptions or special usages)).
For another example of “a tank on the roof”, the system will have the following template:
[tank+roof+RELATIONSHIP]
Or:
[tank+roof+POSITION WITH RESPECT TO THE OTHER OBJECT]
Or:
[tank+roof+on]
Again, the above templates are categorized and stored accordingly, in various (e.g., tagged) hierarchical storages, files, and databases, for future use by the search engine, to dissect, recognize the patterns and templates, and understand the meaning of the sentence or phrase.
In one embodiment, the range of numbers or values or approximate values or measurement accuracies (e.g., length of the table=(5 meter ±2 centimeter)) are expressed based on fuzzy values. In one embodiment, the dimensions in the image (for recognition purposes) are based on approximation, based on fuzzy values.
In one embodiment, the relationships and templates are based on fuzzy terms, with membership values. In one embodiment, the relationships and templates (or grammar) are based on Z-numbers, with terms such as “USUALLY”, expressing concepts such as certainty for the relationships, templates, and grammar.
Multi-Step Recognition:
In one embodiment, the recognition (such as image recognition) is done in multiple steps. For example, for signature recognition, in one embodiment, first, we have a coarse recognition. Then, if the first step of the recognition shows a match possibility, then the system performs the second step of medium recognition. Then, if the second step of the recognition shows a match possibility, then the system performs the third step of fine recognition. Then, if the third step of the recognition shows a match possibility, then the system indicates a match, with corresponding membership value, which is a fuzzy concept. This is a much more efficient method of recognition for most samples and environments (instead of a one-step recognition method). See
For example, for the signature recognition, the first step is the envelop analysis, which is the step of finding the general shape of the signature, and doing the first comparison, to obtain a first degree of match, which is a coarse analysis, as shown in
Xc=(ρiKiXi)/(N(ΣiKi))
where Ki is the weight, value, or intensity for the pixel or image element, and N is an integer denoting the number of pixels, with i as a running variable (an integer, for the summation).
Similarly, for the Y coordinate, we have:
Yc=(ΣiKiYi)/(N(ΣiKi))
This is followed by a second comparison, to obtain a second degree of match, which is a medium degree analysis. Then, the third step, the fine recognition, is to find and compare all pieces of curves and concave and convex shapes in the signature, and map them to an alphabet or dictionary of all typical pieces of curves (cusps or arcs in various shapes, with various angles, ratios, and lengths, and various number of curve or line crossings or loops) and concave and convex shapes (stored in a databases or storage), to convert them in the new language of codes or symbols whose sequence resembles the signature form and shape (as shown in
In one embodiment, a statement describes an event or object, such as a signature's shape, with a qualification of e.g., USUALLY, in the statement, which is a Z-number parameter. Thus, a signature is expressed based on the Z-number.
Context:
The context, for example, can be tagged by the user, or voted by community, or based on history, habit of the user, use of other words, keywords as a flag, or proximity of the words, or any combination of the above. The context (as an attribute) is also a fuzzy parameter, with membership values. One method of measuring the context (C) is based on the word or letter distance (e.g. number of words or letters or paragraphs or pages or chapters or minutes or seconds, as physical distance in between 2 specific words or as the temporal distance or frequency or period between the usage of 2 specific words), or D, which can be expressed, for example, as:
C=1/D
This means that the closer or shorter the distance, the higher the degree of context or related concept between 2 words or phrases or concepts. Or, in general, it can be written as some dimensionless numbers:
C=(K1/D)+K2
where K1 and K2 are some constants or coefficients.
Or, in another embodiment, we have:
C=exp(−D/D0)
where D0 is some constant or coefficient.
In one embodiment, one adds a constant D1 to the equation above:
C=exp(−D/D0)+D1
The context helps us understand that, for example, the word TANK in an article about military budget (as context) refers to a military hardware with engine (that moves on the ground during the war or military exercise). However, in a plumbing magazine, the word TANK refers to a water or fluid tank, as a container. The frequency counter or histogram (e.g., how many times the word MILITARY appears in the article or magazine) and other similar parameters are attached or tagged to the article or file, as a property or attribute.
Contrast:
In one embodiment, the recognition is based on the parameters representing contrast. For example, in an image, a black line in a drawing is detected based on the contrast between neighboring pixels, e.g., black pixels on a line in a white background. For example, the contrast is described as the difference between intensities or grey scale values or values from 2 neighboring pixels, image units, or data units (e.g., in a sequence of data) (or any other form similar to that):
Contrast=ΔX/X=((X2−X1)/((X2+X1)/2))
Thus, the system analyzes the contrast, as a method of the detection of patterns and features, for recognition of objects or features, e.g., face recognition or voice recognition, which uses pixel intensity contrast or sound frequency (and amplitude) contrast, respectively.
In one embodiment, the search engine works on music or sound or speech or talking pieces or notes, to find or match or compare, for taped e-books, text-to-voice conversions, people's speech, notes, music, sound effects, sound sources, ring tones, movie's music, or the like, e.g., to find a specific corresponding music title or movie title, by just humming or whistling the sound (or imitate the music or notes by mouth, or tapping or beating the table with hand), as the input. The output is all the similar sounds or sequence of notes that resemble the input, extracted and searched from Internet or a music or sound repository. See
In one embodiment, the text or speech has content with more than one language. Thus, it has to be distinguished and separated into pieces, first, before it can be further processed for each language separately, as described elsewhere in this disclosure.
Body Language, Expressions, or Emotions:
In one embodiment, the patterns or sequences of sign language or hand movements or eye or lip or facial or foot or body expressions can be recognized, for emotion recognition or translated or converted to text expressions. In one embodiment, the sensors or tags are attached to the body of the user (e.g., the hand of a user), to record movements and positions of a hand with respect to multiple fixed points or coordinates (with beacons or detectors or signal sources) in the room, so that the movements can be recorded and then later interpreted as emotions (e.g., anger) or expressions, such as sentences, data, commands, sequence of information, or signal, e.g., to be converted to text or voice or computer code or instructions, for a person or computer to receive.
For example, this can be used for hands-free navigation of an aircraft by a pilot, using commands, translated based on her body or facial movements or gestures or shapes, e.g., capturing position of facial features, tracking the features, and speed of movements, based on the typical templates of a face or a hand, in a database, to interpret hand signals (e.g., position of fingers with respect to each other, e.g., to indicate that “The package was received.”) or facial definitions or expressions or signals (e.g., position or angle of nose, lips, eye lid, eye, and eye brows, e.g., indicating anger or smile), or based on templates from a specific user for hand or facial gestures. The commands or codes or transcripts or instructions can be fed into a computer or device for a specific action or result. The pattern recognition (described elsewhere in this disclosure) is used to find or interpret the hand or facial signals or data. The interpretations may be not-definite and has a membership value, which is a fuzzy parameter.
In one embodiment, the search is done on multimedia or movies or videos, with text, tags, and sound track associated with it, which can correlate each findings or recognitions from different components of the multimedia, for more accurate overall combined recognition process. In one embodiment, if a piece of a video or the whole video is repeated, similar, or exact copy, to save the storage space (e.g., for video archiving or referencing purposes), depending on the degree of similarity and degree of importance of the video for the user, which are fuzzy parameters, the system may eliminate full or partial data from the video storage(s). For example, for a video with the subject classified as “not-important”, a second video with the same exact data can be deleted, by the policy enforcer module or device, as there is no need for a backup data, based on the pre-set policy in a database, with thresholds and fuzzy parameters or rules, as explained elsewhere in this disclosure.
This method can be used, for example, for minimizing the size of repository needed for video storage web sites (e.g., YouTube.com), or similarly, for email s or attachments carrying the same or similar content or information, e.g., to open up space and delete the duplicative data or files, on a computer or hard drive or server or memory device(s), for faster data management or faster search through that data.
Rules Engine, Filter/Test and Join Networks:
An embodiment implements a rules engine based using Z-valuation or fuzzy maps. In one embodiment, a set of rules are analyzed and the constituents of the antecedent part of the rules are determined, in order to determine pattern in the antecedent parts among rules. This approach helps dealing with many rules in a system where similar antecedent parts appear within different rules. In this approach, the redundancy in evaluating antecedent parts is eliminated/reduced and the temporal issues and inconsistent evaluations of the same parts in different rules are prevented. In one embodiment, a pattern network nodes based on rules' antecedents is setup, e.g., by filtering the variable attributes used in rules' antecedents. In one embodiment, multiple fact patterns satisfy/trigger/fire the same rule. In one embodiment, the facts or propositions are propagated through a pattern network, and a link or a copy of the fact/proposition (or a part thereof) is associated to a pattern node (or added to an associated list/table) along with a truth value indicating how well the fact satisfies the pattern/test/filter associated with the pattern node. For example, if a pattern associated with a pattern node is (X is A) and the fact propagated is (X is B), then the truth value is determined, for example, based on max-min approach (i.e., maximum, for all x, of minimum of μA(x) and μB(x)). In one embodiment, a join network comprises of join nodes based on antecedents of rules to determine the fact patterns satisfying the antecedents. In one embodiment, the list of facts/working memory from pattern network nodes are joined with other lists of facts/working memory from nodes of pattern network of join network, in order to build up the antecedent or parts of antecedent of each rule, at each node of join network. In one embodiment, the joining is performed via a binding variable in both lists being joined. In one embodiment, the truth value associated with the joined record is determined by the truth values of the joining records and the type of the join. For example, in a conjunctive join the truth value of the joined record is determined as minimum of the truth values of the joining records. In one embodiment, the truth value associated with the joined record is also based on the binding variable matching from records of the lists being joined. For example, in one embodiment, where the binding variable has a fuzzy value in one or both lists, the threshold for binding records from the lists (e.g., in equality test of binding variable) or associated truth value based on the binding is determined based on a max-min approach. For example, if the binding variable has fuzzy values A and B in two lists being joined, then the threshold or binding truth value is determined by maximum, for all x, of minimum of μA(x) and. μB(x). For example, if the binding variable has fuzzy values A and crisp value b in two lists being joined, then the threshold or binding truth value is similarly determined as μA(b).
To illustrate an embodiment, suppose the following example of facts provided to the rules engine or inference engine.
Rob is Vera's son.
Alice is Vera's daughter.
Vera is a woman.
Rob's age is mid twenties.
Alice's age is mid thirties.
Alice is young (with low confidence in accuracy of speaker),
Also, suppose there is a rule indicating:
If a woman is middle-age then <some consequent>.
The facts are presented in a protoform and relationships are setup (e.g., in database or linked memory), as for example, depicted in
Son(Vera) is Rob.
Daughter(Vera) is Alice.
Gender(Vera) is female.
Age(Rob) is *25.
Age(Alice) is *35.
With the rule antecedent being:
(Age(<var1>) is middle-age) and (Gender(<var1>) is female).
In one embodiment, based on the existing attributes and relationships (e.g., age, son, daughter) other attributes and relationships are extracted from an attribute/relationship database based on context and existing attributes. For example, a reciprocity relationship is queried and results are used to expand the relationship between the objects or records. For example, relationships “son” and “daughter” result in the reciprocal relationships “parent” or “mother” or “father” (depending the gender of the parent). In one embodiment, the reciprocal relationships per object/record relationship is further filtered based on the existing attributes of the object/records. For example, reciprocal relationship “father” is filtered, while reciprocal relationship “mother” is kept, based on the value of the gender attribute of object/record “Vera” where the queried relationships “son” and “daughter” are based. In one embodiment, consequential attributes are determined, e.g., by querying an attribute/relationship database. For example, the consequential attribute query of “son” (to “Rob”) results in consequential attribute for “Gender” with value of “male” to object/record “Rob”. Similarly, the consequential attribute query for “daughter” (to “Alice”) results in consequential attribute of “Gender” with value of “female” to object/record “Alice”.
In one embodiment, synonym/linked attributes are queried, and the results are instantiated as supplemental relationships between the objects/records. For example, a query for “son” or “daughter” results in relationship “child”, and in an embodiment, a supplemental “child” relationship between the records “Vera” and “Alice” is instantiated. Similarly, in one embodiment, “parent” relationship from between “Rob” (or “Alice”) to “Vera” is instantiated (not shown in figures), based on equivalence/superset to the corresponding “mother” relationship/attribute.
In one embodiment, additional relationships (e.g., “brother” and “sister” between “Alice” and “Rob” (not depicted in figures)), are determined from knowledge base, by matching a set of related object/attributes to a set of general rule(s) for expanding relations/attributes. For example, in one embodiment, the following rules in knowledge base
parent(<var1>) EQUAL, parent(<var2>)
-
- THEN Bi_Direction_Relation(<var1>, <var2>, Sibling);
IF Sibling(<var1>, <var2>) AND Gender((<var1>) is Male
-
- THEN Relation_To(<var2>, <var1>, Brother);
IF Sibling(<var1>, <var2>) AND Gender(((var1>) is Female
-
- THEN Relation_To(<var2>, <var1>, Sister);
when binding with object/records “Alice” and “Rob”, results in bi-directional Sibling attribute/relationship between “Rob” and “Alice”, directional “Sister” and “Brother” attribute/relationship and/or protoforms.
In one embodiment, parallel/suggestive attributes are queried, e.g., from an attribute/relationship database. For example, a parallel/suggestive query for “Age” attribute, results in attribute “Birth”. In one embodiment, a template set of attributes/relationship is determined based on the result of such query. For example, along with attribute/event “Birth”, other related attributes, e.g., “Time” and “Place” related to “Birth” are returned as set/template for application and instantiation. For example, such template is applied to objects/records “Vera”, “Rob”, and “Alice”, e.g., based on their existing attribute “Age”. In one embodiment, the instantiation of template results in separate records and relationships for each instance. A template may include a class level attribute with instantiation at the class level. In one embodiment, the expanded attributes/relationships are supplemented to the relationships and records, e.g., in database. In one embodiment, a protoform of the existing attributes/relationships are instantiated and/or linked to the objects/records, as for example, depicted in
Mother(Rob) is Vera.
Mother(Alice) is Vera.
Child(Vera) is Rob.
Child(Vera) is Alice.
Gender(Rob) is male.
Gender(Alice) is female.
In one embodiment, placeholder objects/records or protoform fragments are instantiated, Birth(Alice), Time(Birth(Alice)), Place(Birth(Alice)), Birth(Rob), Time(Birth(Rob)), Place(Birth(Rob)), Birth(Vera), Time(Birth(Vera)), and Place(Birth(Vera)). In one embodiment, such fragments or placeholder/records/objects are used to further discover relationships and potential joins.
In one embodiment, a query (e.g., an iterative query) is made to expand the facts and related rules from the knowledgebase. For example, a query into the attributes and records results in the following attributes (as depicted in
G1: Age(Mother(<var1>), at time(birth(<var1>))) is range[*20, *40], most likely.
In this example, <var1>is indicative of instantiation point or join potential.
In one embodiment, as for example depicted in
F1: Age(<var1>, at present (DEFAULT)) is
-
- Age(<var1>, at time(<var2>))+Elapsed(time(<var2>), present DEFAULT));
F2: Age(<var1>, at time(birth(<var1>))) is 0;
F3: IF time(<var2>) is before(time(birth(<var1>)))
-
- THEN (Age(<var1>, at time(<var2>)) is UNDEFINED;
F4: time(<var2>) is after(time(death(<var1>)))
-
- THEN (Age(<var1>, at time(<var2>)) is UNDEFINED;
In one embodiment, the contextual facts/functions are provided as template/set to supplement via instantiation and/or used in bind/join operation. In one embodiment, such instantiation further extends the attributes related to records/objects, as for example depicted in
In one embodiment, to efficiently match the facts and rules, a network (e.g., linked) of objects/attributes/filters and a network of join lists are setup. For example, based on the protoform and attributes list of objects/working memory elements are determined and associated with such attributes or protoforms. For example, protoform “Age(Mother((var1>))” in G1 has a potential match with “Rob” or “Alice” when binding to <var1>, where as “Time(Birth(<var1>))” has potential match with “Rob”, “Alice”, or “Vera”, based on existing records/objects. Joining based on the common value, i.e., by enforcing the consistency of <var1>(e.g., via a database join operation with WHERE clause of JOIN or SELECT statement), results in joining on records “Rob” and “Alice”. In one embodiment, the instantiations of general facts/functions result in additional elements or attributes (e.g., as described above for “Elapse”), in a backward chaining method. For example, in one embodiment, the following function/record/relationship is instantiated, based on F1, via binding of <var1>with “Vera” (having an attribute “Age”) and binding of <var2>with “Birth(Rob)” event/record (having an attribute “time”):
Age(Vera) is Age(Vera, at time(Birth(Rob)))+Elapsed(time(Birth(Rob)));
Similarly, the following is instantiated, in an example:
Age(Vera) is Age(Vera, at time(Birth(Alice)))+Elapsed(time(Birth(Alice)));
In one embodiment, an instantiation results in further supplement of attributes for objects/records, e.g., by scanning the form of the template with binding values and linking to the existing object if it already exists (e.g., Age(Vera)) or instantiating additional attribute/object if not existing (e.g., Elapsed(time(Birth(Rob))) or Elapsed(time(Birth(Alice)))) as for example, depicted in
In one embodiment, the instantiation of the general facts or functions result in further facts that act as functions or facts bridging or aggregating other facts. For example, instantiation of G1, based in binding <var1>with “Rob” and “Alice” due to matching/filtering protoforms (“Age(Mother( ))” and “time(birth( ))”) and joining the result consistent with <var1>, results in:
Age(Mother(Rob), at time(birth(Rob))) is range[*20, *40], most likely.
Age(Mother(Alice), at time(birth(Alice))) is range[*20, *40], most likely.
In one embodiment, protoforms are resolved based on one-to-one or many-to-one type relationships. For example, Mother(Rob) is resolved to Vera or refers to the same record/object. Similarly, Mother(Alice) is resolved to Vera:
Age(Vera, at time(birth(Rob))) is range[*20, *40], most likely.
Age(Vera, at time(birth(Alice))) is range[*20, *40], most likely.
Note that the instantiation of F1 results in additional combinations when joining the list based on common attributes/protoforms. For example, binding of <var1>with “Vera”, “Alice”, and “Rob” (having an attribute “Age”) and binding of <var2>with “Birth(Vera)”, “Birth(Alice)”, and “Birth(Rob)” event/record (having an attribute “time”), creates 9 Cartesian combinations (two mentioned above), e.g.:
Age(Vera) is Age(Vera, at time(Birth(Vera)))+Elapsed(time(Birth(Vera));
Age(Vera) is Age(Vera, at time(Birth(Alice)))+Elapsed(time(Birth(Alice));
Age(Vera) is Age(Vera, at time(Birth(Rob)))+Elapsed(time(Birth(Rob));
Age(Alice) is Age(Alice, at time(Birth(Vera)))+Elapsed(time(Birth(Vera));
Age(Alice) is Age(Alice, at time(Birth(Alice)))+Elapsed(time(Birth(Alice));
Age(Alice) is Age(Alice, at time(Birth(Rob)))+Elapsed(time(Birth(Rob));
Age(Rob) is Age(Rob, at time(Birth(Vera)))+Elapsed(time(Birth(Vera));
Age(Rob) is Age(Rob, at time(Birth(Alice)))+Elapsed(time(Birth(Alice));
Age(Rob) is Age(Rob, at time(Birth(Rob)))+Elapsed(time(Birth(Rob));
In one embodiment, the instantiation of other general facts/rules is used to simplify or evaluate the other facts or relations, e.g., by evaluating or substituting the prototype fragments. For example, instantiating F2 by binding <var1>with “Vera”, “Alice”, and “Rob” (having an attributes “Age” and “time(birth( )”) results in the followings:
Age(Vera, at time(birth(Vera))) is 0;
Age(Alice, at time(birth(Alice))) is 0;
Age(Rob, at time(birth(Rob))) is 0;
In one embodiment, the relationships are partially or iteratively evaluated, e.g., by simplifying the protoforms by substitution or by creating relationships. For example, based on instantiation of F2, several of F1 instances become:
Age(Vera) is Elapsed(time(Birth(Vera));
Age(Alice) is Elapsed(time(Birth(Alice));
Age(Rob) is Elapsed(time(Birth(Rob));
In an embodiment, additional relationships/attributes are made between records/objects based on the evaluations. For example, as depicted in
In one embodiment, such simplification is done at the template/class/general functions/rule level. For example, in one embodiment, general facts are joined via binding variables having common attributes. For example, general facts F1 and F2 are joined based on F1:<var2.>and F2:birth(<var1>) both having “time( )” attribute, resulting in a general fact that:
F1′: Age(<var1>, at present (default)) is Elapsed(time(birth(<var1>)), present (default));
In one embodiment, additional general facts are derived based on other facts via a background process. In one embodiment, the additional facts are tested against specific test scenarios for scoring and validations. In one embodiment, additional facts are promoted/tagged as general facts after a validation process and/or passing a validation threshold.
In one embodiment, the instantiation of other general facts/rules is used to filter and trim inapplicable combinations. For example, the instantiation of 13 with binding of <var1>with “Vera”, “Alice”, and “Rob” (having an attribute “time(birth( )”) and binding of <var2>with “Birth(Vera)”, “Birth(Alice)”, and “Birth(Rob)” event/record (having an attribute “time”), creates 9 Cartesian combinations, including, e.g., “Birth(Vera)” for <var2>and “Rob” for <var1>:
IF time(Birth(Vera)) is before(time(birth(Rob)))
-
- THEN (Age(Rob, at time(Birth(Vera))) is UNDEFINED;
For example, further evaluation (e.g., in a subsequent cycle or during a instantiation of a general fact by feeding the instance through a filter/test network) of this instance (e.g., using other generalized facts/functions), trims and nullifies the following G1 instance:
Age(Rob) is Age(Rob, at time(Birth(Vera)))+Elapsed(time(Birth(Vera));
given that Age(Rob, at time(Birth(Vera))) is evaluated as UNDEFINED.
Other instances of F1, for example, are further simplified/substituted or used to build further relationships (based on other instantiations of E1), e.g.:
Age(Vera) is Age(Vera, at time(Birth(Alice))) Age(Alice);
Age(Vera) is Age(Vera, at time(Birth(Rob)))+Age (Rob);
In one embodiment, a candidate generalized fact is generated (e.g., in protoform) based on instantiated/modified facts, e.g., by resolving multiple object references to the same object. For example, in one embodiment, from the above statements, one or more of the following candidate generalized facts are obtained:
Age(<var1>) is Age(<var1>, at time(Birth(child<var1>)))+Age(child<var1>);
Age(mother(<var1>)) is Age(mother(<var1>), at time(Birth(<var1>)))+Age (<var1>);
In one embodiment, as for example depicted in
In one embodiment, multiple valuation of a record/object (e.g., Age(Vera)) is aggregated by aggregating test scores related to the valuations. (For example, see more detail in section Scoring with Fuzzy Map and
In one embodiment, a set of candidate probability/statistical distribution is instantiated per object/record having Z-valuation, e.g., Age(Vera, at time(Birth(Rob))) and Age(Vera, at time(Birth(Alice))) both valued to (range[*20, *40], most likely), are associated each to a set of probability/statistical distribution candidates. In one embodiment, a set of test scores are associated/instantiated per object/record having Z-valuation. In one embodiment, the candidate probability distributions are scored based on facts/rules/functions related to a specific record/object with the resulting test scores associated to the specific corresponding record/object. In one embodiment, the candidate probability distributions are shared by same Z-valuations, while the corresponding test scores are associated to specific records/objects based on the facts/rules/functions related to those specific records/objects. For example, in applying the following fact/function
Age (<var1>) is Age(mother(<var1>))−Age(mother(<var1>), at time(Birth(<var1>)));
to “Rob” and “Alice” by binding to <var1>, aggregate functions affecting Age(Rob) and Age(Alice) are obtained, for example:
Age (Rob) is Age(Vera)−Age(Vera, at time(Birth(Rob)));
Age (Alice) is Age(Vera)−Age(Vera, at time(Birth(Alice)));
For example, in one embodiment, a set of probability distribution candidates are set up for variable representing Age (Rob), and test scores are determined, for example, via Z-valuations imposed via Age(Vera, at time(Birth(Rob))) (i.e., range[*20, *40], most likely). Such test scores alone are expected to be the same as those for a set of probability distribution candidates set up for variable representing Age (Alice). However, the application of other facts to the scoring of the probability distributions, in one embodiment, results in different scoring (aggregate) per record/object/variable. For example, facts (Age(Rob) is min-twenties) and (Age(Alice) is mid-thirties) produce different scores for the same set of probability distributions (pi), i.e., (pi·μMid-20s) score is in general different from (pi, μMid-30s) score. In one embodiment, the resulting aggregate test scores associated with the candidate probability distributions of the same Z-valuations are different and are associated with specific records/objects (e.g., Age(Rob) and Age(Alice)).
In one embodiment, as mentioned above, supplemental facts (specific or general) are determined by applying a template for equivalent transformation (e.g., including substitutions) to recognized protoforms. For example, in one embodiment, querying form (A is B+C) in a knowledge database results in a set of equivalent templates including (B is A−C) or (C is A−B). Applying the equivalent forms, for example, by parsing and substitution or reference to objects, generates and expands the facts base or aggregate function sets.
Join Operation:
In one embodiment, the joining of the lists is optimized by using the ordering or indexing on the lists. In one embodiment, the crisp and fuzzy values of X in a list are ordered based on partial ordering <, e.g., based on alpha cuts and interval comparison. In one embodiment, as shown in
In one embodiment, as for example depicted in
In one embodiment, tagging marking (e.g., for removal) is done via codes and/or attributes associated with items in auxiliary queue/list/table. In one embodiment, several steps are taken in batch mode or at page level, for example, to enhance speed or the database performance. In one embodiment, a positive testing for join is followed by inserting the joined record (from two lists) in a joining list/table or a result set.
In various embodiments, various methods to join lists/tables based on variable taking fuzzy values are used. The resulting joined record, in one embodiment, includes a score relating to the joining values (fuzzy or crisp). For example, when determining the score for joining record associated with A2 from 12105 to Bi from 12110, the test score for the join (or threshold) is for example, determined by max-min approach, i.e.,
In one embodiment, the join test score is used to affect the overall truth value or test score for the joined record, e.g.:
TSjoined record=TSA
Scoring with Fuzzy Map:
In one embodiment, a fuzzy map A* (e.g., as depicted in
For ∀x, α:μA
where c is the possibility degree (or color/grayscale) of the value of membership function. For example, as depicted in
In one embodiment, a similarity measure between A and Aα is used as the basis for color/grayscale distribution with B. For example, in one embodiment as depicted in
In one embodiment, such similarity measure is based with the certainty measure B to determine the possibility measure (i.e., the color or grayscale) for Aα. For example, in an embodiment, as depicted in
cα,B=μB(SIM(A,Aα))
In one embodiment, certainty measure B is expressed as a crisp percentage Bc (as opposed to a fuzzy value). In an embodiment, a fuzzy set Bf is setup based on Bc, e.g., as depicted in
In one embodiment, a non-commutative function of (A, Aα) is used to determine a similarity measure. In one embodiment, a different similarity function is used for α′ (e.g., <α0) than α (e.g., >α0). In one embodiment, a different color/grayscale assignment is used for α′ (e.g., <α0) than α (e.g., >a0). In one embodiment, for example, increasing α (>α0) results in Aα allowing more possibilities, i.e., μAα(x)≥μA(x) for all x, and decreasing α (<α0) results in Aα allowing less possibilities, i.e., μAα(x)≤μA(x) for all x.
In one embodiment, when a fuzzy map, e.g., A*, is used in a calculation, a set {Aα} with corresponding color set c(α,B) is used to determine the result of the calculation. In one embodiment, multiple values of α's are used to model A*. In one embodiment, values of α span the shape of c(α,B). In one embodiment, a predefined number of α's are used to form set {Aα}. In one embodiment, the values of α's corresponding to the significant points of c(α,B) are used to form set {Aα}. For example, in such an embodiment, the corner points of c(α,B) (depicted in
In one embodiment, a fuzzy probability measure (p*) of fuzzy map A*, given probability distribution p(x), is determined using set {Aα}, as follows:
where μp* is the membership function of the fuzzy probability measure p*. In another words, s indicates the possible probability measures of Au, and the color associated with Aα is associated to s as the measure of this possibility (or rather maximum color for various Aα's resulting in the same probability measures is associated with s) indicating the membership function of p* in s domain.
For example, as depicted in
In one embodiment, a test score is associated with a proposition or fact (e.g., in form of X is A). In one embodiment, this test score is based on a probability measure of A based on a probability distribution in X. In one embodiment, a fuzzy test score is associated with a proposition or fact (e.g., in form of X is A*), where the test score is based on a fuzzy probability measure of A* and a probability distribution in X. In one embodiment, multiple candidate probability distributions are used to determine test scores associated with each candidate probability distribution per one or more facts or propositions. In one embodiment, an aggregate test score is determined per candidate probability distribution based on associated test scores based on multiple facts or propositions. For example, as depicted in
ti=( . . . ∧pi,j* ∧ . . . ∧tsi,q ∧ . . . ∧si,k ∧ . . . )
In one embodiment, ̂ operation takes the minimum of all the crisp test scores such as tsi,q and si,k. In one embodiment, the ̂ operation with fuzzy set/numbers (e.g., pi,j*) uses extension principle. In one embodiment, the ̂ operation with fuzzy set/numbers (e.g., pi,j*) uses alpha-cut approach to determine a minimum fuzzy set. In one embodiment, a crisp number is modeled as a discrete impulse having a membership function of one, e.g., as depicted in
Note that usage of “MIN” and “min” are context dependent. For example, in above “MIN” is used to indicate hierarchy/order between two or more fuzzy values/sets, such as “small”, “medium”, and “large”. “min” has been used to indicate the minimum of two values, such as the membership functions values at a given x, e.g., min(μA(x), μB(x)) for all x, for example, to indicate the membership function of (A ∩B).
More Examples & Applications:
In one embodiment, we have a method for fuzzy logic control, in which an input module receives a precisiated proposition associated with a protoform. A fuzzy logic inference engine evaluates a first fuzzy logic rule from the fuzzy logic rule repository. The fuzzy logic inference engine is in or loaded on or executed on or implemented in a computing device, which comprises one or more of following: computer, processor device, integrated circuit, microprocessor, or server. The fuzzy logic rule repository comprises one or more fuzzy logic rules. The fuzzy logic rule comprises an antecedent part and a consequent part. The precisiated proposition comprises a Z-valuation, which is in a form of ordered triple (X, A, B), representing a statement assignment of X to a pair (A, B), where X represents a variable, A is a fuzzy logic set in domain of X, and B is a fuzzy logic set representing a certainty indicator of X being probabilistically restricted by the fuzzy logic set A.
The evaluating step comprises a test score evaluation module assigning a first test score to a candidate probability distribution for X based on the Z-valuation. The candidate probability distribution belongs to a set of candidate probability distributions. The test score evaluation module assigns a second test score to the antecedent part based on the antecedent part, set of candidate probability distributions, and the first test score. The fuzzy logic inference engine determines whether the antecedent part is satisfied beyond a threshold, based on the second test score.
In one embodiment, we have the precisiated proposition comprising a Z-valuation. In one embodiment, we have the consequent part comprising a Z-valuation. The fuzzy logic inference engine determines whether the antecedent part is satisfied beyond a threshold. The system correlates the consequent part with a first truth value based on the antecedent part. The system assigns a first test score to a candidate probability distribution for X based on the Z-valuation. The candidate probability distribution belongs to a set of candidate probability distributions. The correlating step uses the first truth value and the first test score. The fuzzy logic inference engine aggregates a possibilistic restriction on the candidate probability distribution, based on the correlated consequent part.
In one embodiment, we have all parts of the system comprising a Z-valuation. In one embodiment, we have the fuzzy logic rule repository comprising one or more databases, tables, or codes (e.g., as instructions or executables). In one embodiment, the set of candidate probability distributions is generated dynamically, obtained from a database, or input from an interface, e.g., by a user. In one embodiment, the set of candidate probability distributions is based on one or more parameters associated to a model of probability distribution function, e.g., a family of class of probability distribution functions. In one embodiment, the fuzzy logic inference engine uses backward chaining inference or forward chaining inference. In one embodiment, the fuzzy logic inference engine uses a pattern matching algorithm in a forward chaining inference. In one embodiment, the fuzzy logic inference engine performs one or more join operations with variable binding.
In one embodiment, the system comprises a rule execution or a rule firing manager, an agenda or task manager, a knowledge base database or storage, a parallel rule execution module, device, or subsystem, a goal analyzing module or device, a resolving module or device, a defuzzification module or device, an aggregation module or device, a correlation module or device, and/or a join network. In one embodiment, the fuzzy logic inference engine comprises the test score evaluation module. In one embodiment, the fuzzy logic inference engine is separate or different from the test score evaluation module.
Specific Applications:
In different embodiments, the system is designed for the different applications, such as:
-
- (a) economics and stock market or decision analysis (see
FIG. 94 ), - (b) risk assessment and insurance (see
FIG. 95 ), - (c) prediction or anticipation (see
FIG. 96 ), - (d) rule-based characterization of imprecise functions and relations (see
FIG. 97 ), - (e) biomedicine and medical diagnosis (see
FIG. 99 , e.g., for tele-medicine and remote diagnosis), - (f) medical equipment and measurements (see
FIG. 98 , e.g., for measuring blood pressure or X-ray analysis), - (g) robotics (see
FIG. 100 , e.g., on a factory floor for an assembly line), - (h) automobile (see
FIG. 101 , e.g., measuring environmental parameters, to adjust braking system in different driving conditions), - (i) control systems and autonomous systems (see
FIG. 102 , e.g., for driving a car autonomously, without a driver), - (j) searching for objects, search engines, and data mining (see
FIG. 103 , e.g., for searching to find friends in the vicinity of the user (or the store), for social networking, event planning, or marketing purposes), - (k) speaker or voice recognition (see
FIG. 104 , for an example of a voice recognition system), - (l) pattern or target recognition (e.g., airplane recognition or detection, or tracking in video frames, with signature or main features for an airplane) (see
FIG. 105 ), - (m)security and biometrics (see
FIG. 106 ), - (n) translation between languages (For example, one can use multiple systems for interpretation as shown as a part of
FIG. 72 , with one system per language, feeding each other, as a cascade, to translate between languages.).
- (a) economics and stock market or decision analysis (see
In one embodiment, the system does the translation between 2 languages, however, there is not a one-to-one mapping or relationship between 2 words or phrases in the 2 languages. Thus, the system uses the context to find the proper meaning, and for the second language (to which it is translated), the system carries the real meaning as an attachment to the word. For example, for the second language, for the translated part, we have:
[Tank, CONTAINER]
where TANK is the translation in English, and CONTAINER is the real concept behind the word TANK, to remove the ambiguity in the translation (as the word TANK has at least 2 meanings in the American English language).
Surveys:
In one embodiment, the system collects data through voting, survey, on-line, on-paper, using experts, using psychologists, using linguists, collecting opinions, with question on multiple choices with degree of agreement e.g., between 0 to 100, telephone surveys, computer surveys, online surveys, using social networks, using databases, government surveys, random surveys, statistical analysis, population specific surveys, target specific surveys, market surveys, using market reports, using census data, using agents on Internet, using robots, using search engines, or using neural networks as trainers, in order to get membership values, meaning of words or phrases in a language, region, dialect, profession, city, country, or population, language dynamics and evolvement, new words or usage of words, new technical words or Hollywood words or slangs, find the rate of changes in meanings, convergence or divergence of words or concepts or usages, define or extract membership curves and functions, reliability, credibility degree or value, information value, trustworthiness of the speaker or source, or any fuzzy parameter or Z-number concept, e.g., those defined or used in this disclosure.
This is a time-dependent exercise and concept, and it must be updated, as needed, or regularly, depending on the degree of dynamics of the vocabulary or dictionary or slangs or culture or industry or concept or immigration or changes in population mix, which are fuzzy values by themselves. The results of surveys and opinions of people, users, experts, section of population, and other data are stored in databases for future use, for example, for definition or values for Fuzzy membership functions or Z-number interpretations and applications.
In one embodiment, the system handles multiple Z-valuations or numbers. In one embodiment, the system does the reasoning step and/or summarization step with Z-valuations or numbers.
In one embodiment, please note that there are two types of IF-THEN statements. For the first type, at the THEN part, we set a value for a variable. Thus, if the IF section is partially satisfied, based on a membership value, then the value of the variable can be clipped or scaled down (e.g., as a ratio) based on (e.g., proportional to) the membership value. For the second type, at the THEN part, we have an action, e.g., to turn off the light switch for an equipment, which is a binary decision. In this case, if the IF section is partially satisfied, based on a membership value, then we have a threshold(s) (or ranges of values), for which for the values above or below the threshold, to activate or fire the THEN part, e.g., turn off the light switch for an equipment. The threshold can be expressed based on an absolute value, a relative value, a range, a Z-number, or a fuzzy value. Examples of threshold are 0.1, 0.5, 10 percent, 10 percent of average, 10 percent of maximum value, open/close range of real numbers (0, 0.5], 10 Kg (i.e. kilograms, for mass measurement), “usually 10 years”, or “about 10 years”.
Please note that since our method of computation is the closest to the human thinking and speech, it would be the most efficient way of instructing the machines to do a function based on the user's voice command (after parsing the speech, for speech recognition, and conversion to text, commands, templates, or computer codes, based on pre-defined and dynamic/adjustable grammar or rules).
Control systems, e.g., with multiple (If . . . Then . . . rules, can be used for efficient washing machines (consuming less water and detergent, based on level of dirt and type of clothing), braking system for train or cars (for optimum braking), air-conditioning system (better control of the temperature in the room, with less waste in energy), cameras or copy machines (for better image color adjustment or contrast adjustment or ink concentration), car fuel injection systems (for better air and fuel supply, for different engine environments and performances), parallel parking or autonomous driving cars (for optimum performances), robots in a factory assembly floor (with variations on objects received, on the manufacturing steps, for optimum correctional procedures), self-diagnosis and self-repair robots (for best possible diagnosis, to fix itself), system-of-systems (e.g., a colony of swimming robots acting together for a common task, e.g., finding an object in or under water, for proper target recognition or classification and proper feedback to each other, to guide other robots to proper areas of the ocean floor, to avoid duplicative work and effort by other robots in the colony), or any operation of complex machinery in a complex environment for optimum results. (The rules are discussed elsewhere in this disclosure.)
In one embodiment, one uses composite maximum for the defuzzification step. In another embodiment, one uses composite moments (for the area under the curve, or the center of mass) for the defuzzification step.
For backward chaining inference engine, one can use a system as shown in
As mentioned above, fuzzy reasoning systems can gather knowledge from multiple sources (experts), e.g., conflicting, collaborating, and cooperating experts. In a conventional system, one can use a weighted (biased) average technique, to assign weights on different advisors or sources of information. In the fuzzy system, one can use an adaptive peer ranking parameter (with peer ranking amplification), while firing rules in the fuzzy investment model, and with combination through weighted output averaging, or with combination through fuzzy set aggregation (i.e. combined intelligence). To combine multiple fuzzy models, one uses a system such as the one shown in
A fuzzy cognitive map (FCM) for causal flow can be used for adaptive and feedback systems, to model: if Ai then Aj to Bij, where the nodes are concepts (e.g., Ai and Aj) and Bij represents the degree of strength of the connection between Ai and Aj. To activate each concept, there is an activation threshold required (as the minimum strength required). This diagram can represent complex relationships (e.g., one concept increases or decreases the likelihood of another concept). A fuzzy cognitive map is shown in
For an M-state fuzzy cognitive map, we generally need an M×M matrix for the representation of all the relationships. So, if we get N opinions from N different experts, as N fuzzy cognitive maps, we can combine all N fuzzy cognitive maps using Σ (summation) operation on all corresponding matrix entries (L). Then, if each expert has a different level of expertise or reliability peer or user ranking, or an assigned weight, wj, for j=1, . . . , N), then we will have:
L=Σj (wjLj)
To build a fuzzy model, one can go through iterations, as shown in
For investment portfolio management for a client, one can have a financial management system as shown in
For knowledge mining and rule discovery, one can use Wang-Mendel rule discovery method, to partition input-output spaces into fuzzy regions, then generate fuzzy rules from training data, apply discriminant filter to rules, and create a combined fuzzy associative memory (FAM), which is a matrix (based on the inputs). A method is shown in
With the teachings mentioned above, in one embodiment, one can ask about “the top ten biggest companies” (which may change every year) or “top ten tallest mountains in the world” (which does not change every year), and get an answer by the search engine. See, for example,
The search engine can accumulate data from FACEBOOK or YOUTUBE or social sites or government sites or others on idle times, and store them for future searches in the databases, with classes and sub-classes, for faster retrieval, when needed. That also helps to find or distinguish people with the same exact name, build their profiles, and focus advertisem*nt or marketing products, based on their preferences or past history or behaviors.
Please note that for the teachings above, a function y=f(x) as a graph, but without a known formula, can always be approximated by fuzzy graph, as piecewise approximation on the graph, which makes that relationship fuzzy. Then, one can solve based on the fuzzy graph, instead.
For systems that need load balancing, such as server farms for a search engine company or power generators in a electric grid for a country (which have different down times, delays, redundancies, supplies, demands, growths, expenses, new sources, or the like), the system can work in optimum conditions, or adjust fast, using the fuzzy rules and constraints for the system (as explained elsewhere in this disclosure), e.g., for emergency conditions and procedures, to reduce (for example) the blackout time for the consumers in the power grid in various parts of the country, or e.g., speed up the search engine in all parts of the world (by reducing the demand pressure on some areas, and increasing utilization percentages on idle or under-utilized areas of the server farms, to spread out the computing power in an optimized way), using the fuzzy parameters (such as the utilization factor which has a membership value between 0 and 1), as explained elsewhere in this disclosure.
For databases, the database entries can generally be ordered and compared, with respect to one or more fuzzy rules, to index and sort or extract (or query) some useful information from the database(s), resulting in a listing or an ordered table. For example,
For one embodiment,
The teachings above can be used for speech recognition, as well. For example,
Feature vectors can be used for speech recognition, as well, which can be done after the language is determined. In this case, one tries to match the phones or words with a large database of dictionary of all possible words or phones or sequence of phones in a specific language, pre-recorded and categorized. Again, the membership function values are used to find the possible words, via the possible sequence of phones which make up those words, phrases, or sentences. In one embodiment, the sequence of phones is compared to a chain of pointers connecting database phones, in a predetermined database, for all possible combinations of phones, resulting in all possible words, phrases, or sentences, especially the most common ones in a language, to give a possibility of each candidate word or phrase, to rank and select one or more of them for further processing, depending on some threshold(s), which can be a fuzzy parameter itself. In one embodiment, the sequences of phones are mapped to the words in a relational database, which can be updated by the user frequently, or gets trained to recognize the words (with an accompanied neural network system) for a specific user(s).
The similar teachings can be applied to the OCR (optical character recognition) of typed text or handwriting or signature. The text can be broken down in units of letters, pieces of words or letters, or feature vectors (as a basis for a fuzzy set, corresponding to an N-dimensional feature space), and gets compared with those in a database with variations on style or in handwriting, to find the possible targets, with various membership values.
This can be applied to any pattern recognition system or method, such as image mining or recognition on a large number of images (for example, for satellite or radar or laser or stereo or 3D (3-dimensional) imaging), e.g., using a knowledge-based database, with metadata attached or annotated to each image, identifying the source, parameters, or details of the image, e.g., as keywords or indices (which can also be used for database query). This can be used as a user-trainable search tool, employing a neural network module, with scoring functions using examples and counterexamples histograms. For example, in a bin (or partition) where there are more counterexamples than the number of examples, the resulting score is negative. These can be used for the recognition of (for example) trucks, cars, people, structures, and buildings in the images, with membership values associated with each target recognition. Each stored object or class of objects in the database (of all possible objects) has a signature (or one or more specific features, in an N-dimensional feature space, such as the length of the object, the angle between two lines, or the ratio of the length-to-width of the object), which can be matched to (or compared with) a target, with a corresponding membership value for each feature. This can be used for biometrics and security applications, as well, such as face recognition, iris recognition, hand recognition, or fingerprint recognition (e.g., with feature vectors defined from the curved pieces on fingerprints).
There are 2 major types of fuzzy inference systems: Mamdani-type (using the center of mass of the aggregation result) and Sugeno-type, both of which can be used in the systems of the current invention.
In one embodiment, the fuzzy system is used for trip planning or scheduling and its optimization in a trip or daily work. For example, the time for traffic delays and time for leaving the office, plus the threshold time for catching an air plane, are all expressed as fuzzy parameters, as discussed and analyzed elsewhere in this disclosure.
In one embodiment, when we have many systems, one feeding another one, we may want to keep the result of one in fuzzy form (as fuzzy region(s)), e.g., without applying the centroid defuzzification step. This way, the information does not get lost, when it feeds into another system, and it is also convertible to the human's natural language, based on the combination of predetermined templates and their corresponding hedges, stored beforehand in some database (for comparison and conclusion or conversion).
Context Dependent:
Please note that the concept of “tall” (as an example) is both speaker-dependent and audience-dependent. For example, the same person giving lectures in Holland (having very tall population, in general) and Indonesia means differently, when talking with the audience of different population (having different size and height) in different countries, regarding various concepts, such as “being tall”. This is also time-dependent. For example, if a person is giving lecture in the year 1700 AD (or talk about people living a few hundred years ago), in comparison to today (when people are generally taller), the concept of “being tall” is different for those situations. For some embodiments, the membership function and values are time-dependent. In addition, for some embodiments, the element of time is a part of the context analysis.
General Notes:
In one embodiment, the sum of the values of membership functions (corresponding to any point on the horizontal axis) is exactly 1. See
Please note that for all of our teachings here, different truth-value systems (e.g., those suggested by or known as Lukasiewicz, Godel, Product, and Zadeh), for definitions of e.g., T-norm operation, T-co-norm, and negation, can be used. For example, the symbol means AND, “minimum”, or PRODUCT, for various truth-value systems. We can be consistent on one definition throughout the calculations and analysis (from the beginning to the end), or alternatively, mix the definitions (i.e. use various definitions for the same operation, from various truth-value systems) for various steps of the analysis. Either way, it is covered in our teachings here, for this patent application.
For all the systems taught here, one can use a microprocessor, processor, computer, computing device, controller, CPU, central processing module, processing unit, or controlling unit, to calculate, analyze, convert, and process the data, and it can store the information on a disk, hard drive, memory unit, storage unit, ROM, RAM, optical disc, magnetic unit, memory module, database, flash drive, removable drive, server, PC, RAID, tape, or the like. The information can be processed serially or in parallel. The communication between different units, devices, or modules are done by wire, cable, fiber optics, wirelessly, WiFi, Bluetooth, through network, Internet, copper interconnect, antenna, satellite dish, or the like.
Any variations/combinations of the teachings here/this disclosure are also intended to be covered by this patent application.
Z-Webs:
Here, we introduce Z-webs, including Z-factors and Z-nodes, for the understanding of relationships between objects, subjects, abstract ideas, concepts, or the like, including face, car, images, people, emotions, mood, text, natural language, voice, music, video, locations, formulas, facts, historical data, landmarks, personalities, ownership, family, friends, love, happiness, social behavior, voting behavior, and the like, to be used for many applications in our life, including on the search engine, analytics, Big Data processing, natural language processing, economy forecasting, face recognition, dealing with reliability and certainty, medical diagnosis, pattern recognition, object recognition, biometrics, security analysis, risk analysis, fraud detection, satellite image analysis, machine generated data analysis, machine learning, training samples, extracting data or patterns (from the video, images, and the like), editing video or images, and the like. Z-factors include reliability factor, confidence factor, expertise factor, bias factor, and the like, which is associated with each Z-node in the Z-web.
Approximate Z-Number Evaluation:
In this section, we present a method for approximate evaluation of Z-Numbers, using category sets of probability distributions corresponding to similar certainty measures. All the figures are displayed in Appendix 1, as color images. This is also (partially) the subject of a paper (pages 476-483 of the conf. proceedings) and presentation given at an international Fuzzy conf. in Baku, Azerbaijan, on Dec. 3-5, 2012 (“The 2nd World Conference on Soft Computing”), by the inventors. Appendix 1 is a copy of the paper at the Baku Conf. Appendix 3 is a copy of the VU graph PowerPoint presentation at the Baku Conf. Appendix 2 is a copy of the handwritten notes, in addition to the teachings of Appendices 1 and 3. All the Appendices 1-3 are the teachings of the current inventors, in support of the current disclosure, and are incorporated herein.
A Z-Number is denoted as an ordered pair (A,B), where A and B are fuzzy numbers (typically perception-based and described in natural language), in order to describe the level of certainty or reliability of a fuzzy restriction of a real-valued uncertain variable X in Z-valuation (X,A,B). (See L. A. Zadeh, “A note on Z-numbers,” inform. Sciences, vol. 181, pp. 2923-2932, March 2011.) For example, the proposition “the price of ticket is usually high”, may be expressed as a Z-valuation (price or ticket, high, usually). In Z-valuation, the certainty component B describes the reliability of the possibilistic restriction, R, for the random variable X, where
R(X): X is A (1)
with the reliability restriction given by
Prob(X is A) is B (2)
In another words, the certainty component B, restricts the probability measure of A, denoted by v,
v=Prob(X is A)=∫XμA(x)·px(x)·dx (3)
where μA(x) is the membership function of x in fuzzy set A on X domain, and pX is the probability distribution of X. Therefore, the certainty component B indirectly restricts the possibilities of various (candidate) hidden probability distributions of X by: (eq. 4 below)
where μB(v) is the membership function of the probability measure v in fuzzy set B. Here, we show a method to approximate Z-valuation, based on categories (sets) of pX's with similar probability measures (or resulting in similar certainty measure), as an approach to reuse predetermined calculations of probability measures. First, we demonstrate an example of Z-valuation without such approach, and then, we present an approximate approach to Z-valuation via categorical sets of probability distributions.
A. Z-Valuation: Basics:
The Z-valuation uses the mapping of the test scores given by (4) to each of hidden probability distribution candidates of X (See L. A. Zadeh, “A note on Z-numbers,” Inform. Sciences, vol 181, pp. 2923-2932, March 2011. See also R. Yager, “On Z-valuations using Zadeh's Z-numbers,” Int. J. Intell. Syst., Vol. 27, Issue 3, pp. 259-278, March 2012.), collectively referred to as
Prob. Distrib. Candidates={pi}, (5)
where i numerates different candidates.
vi=μA·pi=∫XμA(x)·pi(x)·dx, (6)
and
tsi=μB(vi). (7)
Note that the dot symbol in (μA·pi) in (6) is used as shorthand for the probability measure.
Via the extension principle, the application of the restriction (test scores) on px,i(x) (i.e., probability distribution candidates in X domain) to other entities is illustrated. For example, the restriction on px,i(x) can be extended to the possibilistic restriction on the corresponding probability distributions, py,i(y), in Y domain, where
Y=f(X),
In such a case, the restrictions can further be extended to the probability measures, wi, of a fuzzy set Ay in Y domain, based on py,i(y). The aggregation of the best test scores for wi would determine the certainty component By in Z-valuation (Y,AY,BY), based on the original Z-valuation (X,AX,BX), as indicated in
For simplicity, as shown in
μB
μB
In other words, in this scenario,
In case of single variable dependency Y=f(X), the probability measure w can be evaluated by unpacking the probability distribution in Y as illustrated by (9) and transforming the integration over X domain as shown in (10), without explicitly evaluating py,i:
where j denotes the consecutive monotonic ranges of f(X) in X domain, and x is the solution for f1(y), if any, within the monotonic range j, for a given y. This takes into account that the probability (py,i ·dy) for an event within the infinitesimal interval of [y, y dy] in Y domain, is the summation of the infinitesimal probabilities from various infinitesimal intervals [xi+dxj] (if applicable) in X domain, where for each j:
dy=f′(xj)·dxj
Therefore, with repacking the integration (9) in X domain over the consecutive monotonic ranges of f(X), we obtain:
wi=∫XμA
Furthermore, if f(X) is monotonic (i.e., f1(y) has only one solution in X, if any) AND μA
Z-Valuation: Example:
To illustrate an example of Z-valuation, assume the following is given:
X=(AX,BX),
Y=f(X)=(X+2)2, and
AY.
The goal is to determine the certainty value BY for the proposition that (Y is AY), i.e., the Z-valuation (Y, AY, BY). For purpose of this example, assume
In this example, the set of candidate probability distribution for X was constructed using Normal distributions with mean (mx) ranging from −2 to 2 and standard deviation (σx) ranging from 0+ (close to Dirac delta function) to 1.2.
By applying (10), we can then determine the probability measure of AY (in Y domain), denoted as w, based on the probability distribution candidates in X domain (i.e., bypassing the direct calculation of the corresponding probability distributions in Y domain). The probability measure w is depicted in
Given that each probability distribution candidate is associated with a possibility restriction test score (as shown for example in
Therefore, to assign a final test score to a given w (or w bin) based on (8), we can determine the maximum test score for all w's associated with the given w bin.
The result of an intermediate step for determining the maximum test score for correlated w's (i.e., falling in the same w bin) is illustrated in
The resulting maximum test score associated with a given w bin defines the membership function of w (or a value of w representing the w bin) in BY, as depicted for this example in
II. Z-Valuation Using Granular Category Sets: A. Predetermined Category Sets: Test Scores, Probability Measures, and Probability Distributions:
The probability measure of AX, denoted as v, may be predetermined and reused, given that the integration in (3) may be normalized based on the general shape of the membership function of Ax and the class/parameters of probability distribution candidates. In normalized form, for example, a category of normalized membership function may be defined as symmetric trapezoid with its support at interval [-1,1] with a single parameter, β, indicating the ratio of its core to its support (as shown in
Furthermore, we may reduce the level and complexity of computation in approximating the Z-valuation by using a granular approach. For example, for a category of normalized AX (e.g., symmetric trapezoid with β of about 0.5, as shown in
Let Vj denote a category/set of probability measures of AX (e.g., probability measure “High”), where j numerates such categories in v domain. Each Vj corresponds to a range or (fuzzy or crisp) subset of probability distribution candidates, denoted by Cj whose pi members are defined via the following membership function: (eq. 11, below)
Therefore according to (11), we may predetermine Cj via a similar method of applying test scores to the probability distribution candidates, pi, (as for example shown in
Furthermore, the certainty levels (test scores) ay also be made into granular (fuzzy or crisp) sets TSk, e.g., in order to reduce the complexity of calculation during the aggregation process of Z-valuation. Index k numerates these test score category sets.
In one approach, the certainty component BX is granularly decomposed or mapped (or approximately expressed) via pairs of probability measure and test score category sets, i.e., (Vj,TSk)'s, as for example demonstrated in
In one scenario, the decomposition of BX may be expressed as series of tuples in the form (Vi,TSk, weightj,k) or simply as a matrix with weightj,k as its elements. Given the correspondence between Cj and Vj, the granular test score sets TSk's are also associated with granular probability distribution candidate sets, Cj's (with the same weightj,k)
In another approach, a non-categorical test score (e.g., a fuzzy or crisp set) TSj is determined for each Vj (and Cj), e.g., by using extension principle, based on mapping via BX:
B. Computation and Aggregation Via Normalized Categories:
One advantage of reusing the predetermined normalized categories is the reduction in number of calculations, such as the integration or summation in determining probability measures per individual probability distribution candidates in X domain or their corresponding probability distributions in Y domain, per (4) and (8). In addition, instead of propagating the test scores via an individual probability distribution candidate, the extension of the test scores may be done at a more granular level of the probability distribution candidate subsets, Cj, which are typically far fewer in number than the individual probability distribution candidates. However, the aggregation of test scores for Z-valuation, e.g., for (N,AY, BY), will involve additional overlap determination involving various normalized category sets, as described below.
The normalization of symmetrical trapezoid membership function AY, e.g., “Y is about nine,” as shown in
Note that the predetermined categorical subset of probability distributions in Y domain, denoted as CY,j, that is associated with Vj, may be distinct from the corresponding one in X domain, denoted as CX,j, e.g., due to parameters such as β (or the class of the membership, such as trapezoid or ramp). For example,
i) Mapping in X Domain:
In one approach to estimate (10), we may determine (or approximate) μA
where pX,iT represents the transformed probability distribution candidate (in X domain) from PX,i.
Since in our example, (depicted in
pX,iT=T(pX,i) =Tt·Ts·pX,i, (14)
with
Ts·PX,i=Ts(mX,i, σX,i)=(s·mX,i, s·σX,i),
Tt·pX,i=Tt(mX,i, σX,i)=(mX,i+t, σX,i).
Once normalized, μA
where Ij,n describes a grade for overlap between CX,nT and CY,j.
For example, as shown in
The determination of the test scores for VY,j may be implemented via a set of fuzzy rules linking CX,a and CY,j. For example, the antecedent of each rule is triggered if the corresponding Ij,n is above an overlap threshold, and the consequent of the rule assigns TSX,k's (or an aggregate of TSX,k's based on weightn,k for a given n) to a variable SCY,j. A simpler test score assignment rule may use a non-categorical test score TSX,a which is determined for each e.g., via (12), based on the mapping through BX:
Rulej,n: if (Ij,n) then (SCY,j is TSX,n) (16)
However, in correlation/aggregation of assigned (fuzzy) test scores to variable SCY,j, we must consider the maximization of test score required by (8). For example, in aggregating the rules for SCY,j, we may use α-cuts to determine an aggregated (fuzzy) result, denoted as AGSCY,j, as follows: (Eq. 17 below)
where Correl(Ij,n, TSn) modifies the membership function of TSX,n by correlating it with the factor Ij,n, e.g., via scaling or truncation. Membership function of BY is then approximated by a series of fuzzy relations (VY,j, AGSCY,j).
For a given w (probability measure of AY), μBY(w) may be approximated as a fuzzy number (or a defuzzified value), by further aggregation using fuzzy relations (VY,j, AGSCY,j), e.g.: (Eq. 18 below)
ii) Overlap Approximation:
An approach to approximate or render the overlap (15) between the category sets, such as CX,n, may use α-cuts to present each crisp α-cuts of predetermined category set as a set of points in (m,σ) space. These sets of points may be modeled efficiently, e.g., based on graphical models, optimized for fast transformation and intersection operations. For example, the models that use peripheral description for the α-cuts allow robust and efficient determination of intersection and avoid the need to transform all the points within the set individually, in order to reduce the computation involved in (13).
iii) Estimation Using Contour Approach:
In addition to predetermining CX,n, based on VX,n, for a normalized set AX, we can predetermine various α-cuts of probability measures (e.g., depicted as contours of constant v in
Another approach uses (e.g., piecewise) representation of BX (not predefined) where based on inspection or description, key values of v associated with key values of test scores may readily be ascertained (e.g., based on α-cuts), resulting in a set of (vi,tsi) pairs. Then, the predetermine α-cuts of probability measures (e.g., depicted as contours of constant v in
For quick estimation of BY in an alternate approach, the predetermined α-cuts (i.e., w's) of probability measures for normalized AY may be used (similar to those shown in
In one embodiment, for Z-number analysis, for probability distributions analysis, the predetermined categories of hidden probability distribution candidates and normalized Fuzzy membership functions facilitate the pre-calculation of probability measures and their associated reliability measures in Z evaluation or as Z-factors, for fast determination of the reliability levels of new propositions or conclusions. This approach opens the door to the extension of the reliability measures (e.g., via extension principle) to new propositions, based on graphical analysis of contours (a-cuts) of similar probability measures in the domain of parameters representing the probability distribution candidates. Basically, we will use the transformation and mapping of categorical set of the probability distribution candidates (represented as regions or α-cut contours) for extension of the reliability measures. This way, as we pre-calculate and store the shapes and results in our library or database for future use (as templates), the new analysis on any new data can be much faster, because we can readily match it with one of the templates, whose results are already calculated and stored, for immediate use.
Now, let's look at Appendix 2. In one embodiment, referring to the top FIG. and derivation on page 1 of Appendix 2, we have different values of Vα,n, based on various α-cuts (with (ts=α)). Then, we match against category (singleton) vs (see the bottom FIG. on page 1 of Appendix 2). Then, on FIG. and derivation on page 2 of our Appendix 2, we get a series of the curves. We use the predetermined contours of probability measures vs,m. Note that (vs,m=pi·μAXnormalized). Note that pi's define the contour(s) for vs,m (or regions of pi's) defining region(s) for vs,m (such as 0 or 1), to interpolate and determine contours (or regions) of constant denoted by Cα,m. These are associated with test scores set by α, i.e. (ts=α) for Cα,m.
Then, on FIG. and derivation on page 3 of our Appendix 2, we transform or do other manipulations, according to extension rules (e.g., on normalized) for μAY:
Cα,mT=T(Cα,m)
While maintaining the test score for Cα,mT(as α). Based on categories of ws,j (similar to vs,n, except for w), probability measure of AY in Y-domain, where ws,j are singletons (predefined), have corresponding contours (or regions) Cs,j (see the figure on the bottom of page 3 of our Appendix 2). Then, we find the intercepts between Cα,mT and Cs,j, if any, i.e. Iα,m,j.
Then, on FIG. and derivation on page 4 of our Appendix 2, based on the intercepts, we find the best test score for a given Cs,j extended from Cα,mT, e.g.:
tss,j=sup∀α′ α′
where Iα′,m,j exists.
(i.e., the best test score from intercept points to a given Cs,j.)
Now, we associate tss,j to ws,j to construct (μBY (w)), and interpolate for other (see the figure on the bottom of page 4 of our Appendix 2). Since tsi,j's source is α, tss,j's appear as α-cuts in μBY, as well.
Then, on derivation on page 5 of our Appendix 2, we have: Where the scenario involves e.g. z=f(x,y), instead of y=f(x) (where the solution may be worked out in the X-domain), we can still use contours (or regions) of specific test scores (e.g., based on α-cuts), and contours determined by interpolation of predefined or predetermined probability measure contours or regions. The manipulation, e.g., (pz=pxOpy), can be implemented based on contours or regions of constant test scores (for X or Y), instead of individual px,i and py,i, to reduce the number of combinations and calculation. The test scores can be extracted from X, Y domains to Z domain (in this example) and maximized based on the intercept points in I), domain with predetermined contours of probability measures of (normalized) AZ, to again calculate μBZ.
Thus, in summary, the above section provides the methods for approximation or calculation or manipulation of Z-numbers, and related concepts. Now, we explain other components of our inventions, below.
Thumbnail Transformation:
In one embodiment, the input data (e.g., image) is preprocessed. For example, the image is transformed into a smaller thumbnail that preserve the high level nature of the image content, while not necessarily preserving its unique characteristics. This may be achieved, for example, by down sampling or aggregation of neighboring pixels. Other methods may include reduction of the variable space by consolidating the colors into intensity (e.g., gray scale) and/or reducing the number of bits representing color or intensity. Such a transformation is denoted as thumbnail.
A thumbnail includes less resolution and data, and hence, it contains less overall detailed features. The purpose is to simplify the task of dealing with many pixels while still managing to detect the high level features associated with the images (or other type of data). For example, using a thumbnail, a recognition module quickly identifies the presence of a head or face (while not intended to necessarily determine the identity of the person or object).
One embodiment uses a preliminary search to detect main features in a thumbnail data/image for fast computation. In one embodiment, the limitation may be on the number of pixels on the visual layer (via preprocessing). In one embodiment, the limitation is imposed on the detection/classifier network (e.g., on hidden layers) itself. For example, the main features are learned and isolated (e.g., by units or neurons of higher hidden layers) or learned by targeted attempt (e.g., by keeping all other weights and letting the weight on certain units change when learning a certain feature.)
Feature Detection and Learning:
In one embodiment, for example where labeled training samples may be difficult to prepare or scarce, the training is done with unlabeled samples to learn the features from the sample details. For example, a restricted Boltzmann machine (RBM) may be used to successively learn the features one layer at a time.
A Boltzmann machine refers to a type of stochastic recurrent neural network, where the probability of the state is based on an energy function defined based on the weights/biases associated with the units and the state of such units. In a Boltzmann machine, some units are denoted visible where the state may be set/clamped or observed and others may be hidden (e.g., those used for determining features). In the Restricted Boltzmann machine (RBM), the weights between hidden units within the same layer are eliminated to simplify the learning process. The learning process tends modifies the weights and biases so that the energy state associated with the samples learned are lowered and the probability of such states is increased. In one embodiment, the state of hidden layers are presented by a stochastic binary variable (e.g., in [0, 1] range) based on a sigmoid such as logistic function. In one embodiment, the energy function is given as
where vi and hi denote the state of the ith visible unit and the jth hidden unit (as for example depicted in
where Eα is the energy associated with state α; T denotes the “Temperature” of the system; the denominator denotes the “partition function”, Z; and β denotes any state of the system. Since the energy of a state is proportional to negative log probability of the state, the probability that a binary stochastic unit j is at state 1 (or ON) in such RBI becomes the following logistic function:
where T controls relative width of the above logistic function, and ΔEj (for example for a hidden unit) is given by:
Note that in an embodiment with T is set to zero, the stochastic nature of the binary units becomes deterministic, i.e., taking the value sigmoid function (zero or one), as in Hopfield Network.
In one embodiment, the training attempts to reduce the Kullback-Leibler divergence, G, between the distributions of V states based on the training sets and based on thermal equilibrium of the Boltzmann machine, by modifying weights and biases, e.g., via a gradient decent over G with respect to a given weight or bias. The aim of training is to determine weights/biases such that the training samples have high probability. In maximizing the average probability of a state V, P(V), with respect to weights, we have
where the average over the data means average over the training data(i.e., when V units sample from the training sets and are clamped to a training sample while hidden units are updated repeatedly to reach equilibrium distribution), and the average over model means the average from Boltzmann machine sampling from its equilibrium distribution (at a given T). In one embodiment, learning algorithm uses a small learning rate with the above to perform gradient decent. Similarly, the following can be used in learning bias cj:
In one embodiment, where the weights are absent between the hidden units, the updating of the hidden states, H, is done in parallel as the hidden units are conditionally independent for a given set of visible states, V. In one embodiment, sampling from model involves one or more iterations alternating between updating (in parallel) hidden and visible layers based on each other. In one embodiment, sampling for the model is substituted with sampling from reconstruction, which updates the hidden units (for example, in parallel) using the visible units clamped to a training set, then updates the visible units (e.g., in parallel) to get a reconstruction from the features in the hidden layers, followed by updating the hidden units based on the reconstruction. This approach approximates the gradient decent of contrastive divergence in an efficient and fast manner. In RBM learning, contrastive divergence can be used instead of maximum likelihood learning which is expensive. In one embodiment, T is lowered from a higher initial value to make low cost (energy) states more probable than high cost states, while the higher initial value of T allows for reaching and sampling equilibrium states quicker. In one embodiment, the stochastic nature of binary units allows escaping from local minima. In one embodiment, during the reconstruction, a subset of visible units are clamped to input data to reconstruct other visible units from the features including those affected or derived (e.g., stochastically) from the input data. The training in such a conditional Boltzmann machine tends to maximize the log probability of the observed visual units (now taken as output in reconstruction), given the input data.
In one embodiment, other non-binary discrete stochastic units may be used. In one embodiment, continuous value units may be used. In one embodiment, mean filed units are used having their state (in the range of [0, 1]) determined by the total input (e.g., a logistic function) and a noise (e.g., as a Gaussian). In one embodiment, other stochastic functions/distributions (e.g., binomial and Poisson) are used for the units. In one embodiment, where continuous data (including semi-continuous data with many levels as opposed to few discrete levels) is used for state of the visible units, the sampling from a probability distribution (e.g., Gaussian with a given variance, with the mean determined by the other signal and weights) keeps the stochastic nature, while making the signal in visible unit continuous (as opposed to discrete). The hidden layers may stay binary (stochastic). In one embodiment, stochastic visible units use continuous signal (e.g., in [0, 1] range) based on other signals and weights and a probability distribution logistic function) for sampling or updating its signal.
In one embodiment, following the training of one RBM, another hidden layer is added on top which employs the lower RBM's hidden layer as input to determine higher level features, and the training is done one layer at the time. For example,
In one embodiment, weights (yi,k) are introduced in the visible layer while training the weights (wi,j) between the visible layer and the hidden layer (e.g., as depicted in
Δwi,j=ε·(vihj0−vihj1)
Δyi,k=ε′·(vivk0−vivk1
where superscript 0 indicates the correlation after the initial update of hidden layer after clamping the training sample to the visual units, and superscript 1 indicates the correlation after the hidden layer is updated next time by the reconstruction at the visual layer. In one embodiment, to get to the reconstruction in the visible layer, the visible units are updated one or more times (e.g., iteratively in parallel) based on the current weights, the updated hidden units, and the state of the visible units (from the initial or prior iteration). In one embodiment, the update activity involves stochastic sampling from the probability distribution (e.g., logistic function). Note that ε and ε′ correspond to the learning rate. In one embodiment, the hidden units are updated multiple times before the correlations are used to determine changes in weight. In one embodiment, visible units with continuous value state (e.g., mean field units) are updated in parallel based on the total input to the unit (e.g., based on a logistic function).
In one embodiment, intra-layer weights are introduced during the training of a higher hidden layer in order to establish tighter relationships among inter-layer units (e.g., neighboring visible units corresponding to neighboring pixels in an image/data). This enforces constraint during generation. In an embodiment, this facilitates the generation of the parts of a larger recognized object that would not fit each other due to loose relationships between corresponding sub-features. In one embodiment, more features (e.g., redundant) are used to tighten the relationships. In one embodiment, the interrelations between the features (e.g., constraints or rules) are used to limit the choices (i.e., placement of parts), and the placement of one feature helps determine the placement of the other features based on the interrelationship between those features.
In one embodiment, as for example depicted in
where vi and vi′ and are the state of the ith units (or intensity of the image at given pixel corresponding to unit i) associated with V and V′, respectively. In one embodiment, for the same number of parameters, deep autoencoders tend to produce less generalization errors compared to shallow ones.
In one embodiment, the dimensionality of the data is reduced via the coding presentation at the coding layer (e.g., H(3)) having few units compared to the number of units in V.
In one embodiment, a noise signal is introduced in the top hidden layer units (e.g., H(3)) during training (but the same for the corresponding training data sample used in V layer) to adjust the weights resulting in more bimodal probabilities in order to make the system more resilient against the noise in the data.
In one embodiment, the features of the training samples are learned, e.g., via an unsupervised learning algorithm (e.g., by greedy learning by RBMs). Then, the features are correlated or associated with labels from a subset of training sample, as for example depicted in
Once the association between the labels and Data is learned, in one embodiment, data is input to V layer, and its corresponding label is ascertained at L layer, by having the units in C layer drive the units in L layer. In one embodiment, data samples corresponding to a label may be constructed by clamping unit(s) in L layer to derive units in C Layer, and followed by a top-down reconstruction in V layer. In one embodiment, a subset of units in V layer are clamped to input (e.g., to input a partial image or a portion of image) and the state of one or more labels are set in L layer by clamping to environment. Then, the other unclamped V units are used to determine the state of the other V units (given the clamped visible and label units), deterministically or stochastically (e.g., through iteration). In one embodiment, a larger image may be recovered from partial data (e.g., partial image) through reconstruction.
Reliability Measure:
In one embodiment, the strength of the correlation between data and label or conformity of data to the system (e.g., a trained system) may be determined based on the energy of states given the clamped data (and label). In one embodiment, the strength of correlation or conformity is based on relative probability of various states. For example, the energy difference of two states in Boltzmann machine (in equilibrium) is proportional to the log of the ratio of their probabilities. In one embodiment, the relative strength of the correlation or conformity is based on the relative probability of two states. In one embodiment, a baseline for the probability of training samples is established during and/or after training. In one embodiment, the strength of correlation or conformity indicates how well the state(s) representing the data (and label) fit into the energy landscape of the system. In one embodiment, as depicted in
In one embodiment, the quality of the search is evaluated based one or more approaches including for example, the probability, e.g., the total energy of RBM, or the difference between the regenerated data/image and the input, the frequency the recognized labels change while anchoring the visible units/neurons to the input/image.
Learning Based on Models:
In one embodiment, the learning is achieved through simulation using a data (and label) sample generation based on one or more models. In one embodiment, a network trained based on model(s) is used to recognize and classify actual data which may not have been seen before. In one embodiment, the system is trained to infer the potential model(s) itself by recognizing the (e.g., observed) data conforming to a particular model and its associated labels/parameters.
In one embodiment, as for example depicted in
In one embodiment, noise is incorporated into the rendering in order to make the network more resilient to noise. In one embodiment, a stochastic noise (e.g., Gaussian) is applied to the rendering, e.g., in illumination, intensity, texture, color, contrast, saturation, edges, scale, angles, perspective, projection, skew, rotation, or twist, across or for portion(s) of the image. In one embodiment, noise is added to a hidden layer in a reproducible manner, i,e., for a given data sample (or for a given model parameters), in order to adjust the weight to result in a more modal range of activities to increase tolerance for noise.
In one embodiment, elastic distortions (as well as affine transformations) are used to expand the size and variety of the training set, e.g., when the training set is produced from a model (such as a rendered data/image) or when the data/image is provided separately as part of a training set. In one embodiment, such a distortion is parameterized and rendered by the rendering unit. One embodiment used both affine (e.g., translation, scaling, reflection, rotation, hom*othety, shear mapping, and squeeze mapping) and distorting type transformations. In one embodiment, various transformations are rendered to generate training dataset to let the system learn features that are transformation invariant. In one embodiment, a shape model is generated with various parameters, such as various textures, colors, sizes and orientations, to let the system learn the invariant features such as the relative positions of the sub features of the modeled shape. In one embodiment, orthogonal matrixes, for example, are used to perform rotation and reflection transformation for rendering the image or on the provided data sample.
In one embodiment, the features of a high level model (with parameters) are learned by a system (such as RBM) through training (e.g., unsupervised). For example, in one embodiment, a 3D model generates various 2D images at different poses (including position, orientation, and scale) and expressions/emotions (or illumination), and the system would learn correlation between the images and their features (derived from the model). Then, the model parameters (and their probabilities) may be obtained for an image.
In one embodiment, various samples are generated/rendered from a 3D model, by varying relative location and angle of the viewer and the model object (e.g., polar coordinates (r, θ, φ)). These variation span various poses (based on θ and φ) and scaling (based on r), using other perspective parameters (e.g., derived from camera/viewer's view span).
In one embodiment, a 3D model rendering mapped to 2D images is based on the normal vectors at a given point of the 3D model, illumination parameters (e.g., location of light(s) and intensity), and reflectivity and texture model of the surface. In one embodiment, the location/presence of rigid points from the model improves the accuracy. In one embodiment, PIE (pose, illumination, expression) variations are used to generate training data/images (e.g., by rendering in 2D).
In one embodiment, multiple models can be learned in combination. E.g., the model for generating of texture of surfaces or colors can be learned in conjunction with a 3D model of head or body. In rendering a 3D model, the texture model may be incorporated to provide textures and colors for the rendered images used for training. The correlation between the model parameters and the rendered images is learned via training. In one embodiment, noise is added to prevent over fitting and regularize the weights to better generalize when used with out of sample data/images.
In one embodiment, getting a low level of conformity of a data/image (for example based in a conformity measure such as energy error or probabilities) with a trained system (e.g., based on a model) causes the data to be marked/tagged or included in a set of data to be recognized/classified by other expert systems/networks.
In one embodiment, the model comprises of rules governing the parameters, structure, and relationships between various components and sub-components of the model. In one embodiment, the rules engine is iteratively executed to generate sample data for training, by using a rules engine.
In one embodiment, the model includes a databases of background and foreground objects (with parameters) or images. In one embodiment, various data samples are created with various background and foreground models to train the system recognize high level features of foreground and background (e.g., wide uniform horizontal bands or regions of color/intensity). In one embodiment, generic labels are used to train the correlation between the labels and the features of the background or foreground scenes.
Correlating of Features and Locations of Interest within the Data (e.g., Image):
In one embodiment, a location within the image is specified by a continuous value (e.g., in range of [0, 1] to indicate/identify the location or pixel along a direction (e.g., x or y direction) in the data/image) or a multi-discrete value (e.g., indicating/identifying a range of locations or pixels along a direction in the date/image). In one embodiment, as for example depicted in
In one embodiment, where inter-layer links between units are not fully connected, the connection from M layers to units in lower hidden layer(s) are substantially arranged to spatially resemble or correspond to M units' corresponding pixels (or range of pixels) in the image viewed via V layer. In such a case, the links from V layer to higher hidden layers are also limited in number of connectivity, and for example, the few links follow a fan out pattern from a 2D layout of V layer to next hidden layer.
In one embodiment, blobs (of fuzzy blobs) are provided on M layer for association with the image during training. Fuzzy blob, for example, may have fractional membership function value at the blob's edge. In an embodiment, the membership function value in range of [0, 1] is represented by a logistic function in a unit.
In one embodiment, the location, area, or focus of interest is provided on M layer with the corresponding training sample in V layer, to train the correlation. In one embodiment, the representation of the focus of interest may be a (fuzzy or crisp) border or a region specified parametrically or per pixel.
In one embodiment, with a training sample having multiple focuses of interest, the training may be performed by submitting the same data (image) with individual focus of interests during the training. In one embodiment, the stochastic nature of C layer will cause reconstruction of focus of interest in M or P layers, given an input image (or a portion of image) in V layer. For example, in training face recognition, images including one or more faces are supplied to V layer while their corresponding focuses of interest (e.g., the location/size of the face) are supplied to M or P layers, to train the correlation. In one embodiment, the various focuses of interest are iteratively constructed in M or P layer by clamping data (e.g., an image) in V to, for example, derive stochastically the corresponding focuses of interest from C layer. In one embodiment, the reconstructed parameters are output in M or P layers based on their corresponding probability.
In one embodiment, the correlation of image/data to its locations of interest is performed during training by implementing a representation of such locations on a layer of units laid out to correspond to the image/data (e.g., by linking such units to a hidden layer above V layer). In one embodiment, the position parameters (e.g., location, width/height, type, orientation) and the coverage parameters (border type, fill type, fuzzy/crisp) are used to render representation of the location(s) of interest on the representation units, e.g., by using a value in range of [0, 1]. in one embodiment, the fuzzy type rendering helps avoid making false correlations with other irrelevant features in the image/data, by representing the features of the location of interest as coarse. Fill type rendering identifies a blob where the location of interest is in the image, so that if the features of the interest are in the middle of the location, the training would catch the correlation.
In one embodiment, we have: a system for image recognition in an image recognition platform, said system comprising: an interface which receives an image; said interface receives a location of interest; a neural network; wherein said neural network comprises a visual layer and a first hidden layer; wherein said visual layer is located below said first hidden layer; wherein said neural network receives said image and said location of interest; wherein said image is connected to said visual layer; a parameter layer; wherein said parameter layer is added to said neural network; a representation layer; wherein said representation layer is added to said neural network; wherein said parameter layer has information for coordinates, width, height, orientation, or type of shape for said location of interest; wherein said representation layer represents a value, values, or range of values that said parameter layer has for said location of interest; wherein said representation layer has a weighted link to a second hidden layer, connected horizontally from side of said neural network; wherein said second hidden layer is located between said visual layer and said first hidden layer; wherein said second hidden layer is located above said visual layer; wherein said second hidden layer is located below said first hidden layer; a correlation layer; wherein said correlation layer is located above said first hidden layer. (Please note that the parameter layer is optional and can be bypassed by layer M (or representation layer), i.e., directly connecting to layer M, in
In one embodiment, we have these options/additions/variations:
wherein said representation layer is connected to said correlation layer in both directions.
wherein said parameter layer is connected to said correlation layer in both directions.
wherein said correlation layer correlates said representation layer with said image.
wherein said correlation layer correlates said parameter layer with said image.
wherein said correlation layer correlates said location of interest with said image, using said representation layer.
wherein said correlation layer correlates said location of interest with said image, using said parameter layer.
wherein said correlation layer reconstructs, in reverse mode, after training.
wherein said system comprises or applies one or more of following: softmax, cross entropy, sigmoid cross entropy, contrastive, Eucledean distance, sum of squares of difference, multinomial logistic, infogain, generalization of multinomial logistic, or hinge or margin loss layer, unit, or comparison module.
wherein said system comprises or applies one of following between said representation layer and said second hidden layer: softmax, cross entropy, sigmoid cross entropy, contrastive, Eucledean distance, sum of squares of difference, multinomial logistic, infogain, generalization of multinomial logistic, or hinge or margin loss layer, unit, or comparison module.
wherein said neural network is not fully connected.
wherein said connection between said representation layer and said second hidden layer is not fully connected.
wherein said neural network comprises convolutional neural network connectivity format.
wherein said representation layer is expressed in Carthesian coordinates.
wherein said representation layer is expressed in polar or angular coordinates.
wherein said parameter layer is expressed in Fuzzy values.
wherein said location of interest is a part of an object represented by said image.
wherein said location of interest is represented as a coarse value or Fuzzy value.
wherein said system is used or applied recursively in said image recognition platform, to find or distinguish or detect or recognize various objects and their components. (See
Limiting Number of Weights Based on 2D Fan Out Layout:
In one embodiment, as for example depicted in
In one embodiment, there are multiple type of units in a hidden layer (e.g., H(1), with each type corresponding to different number (or range of number) of links to its lower layer units. In one embodiment, for example, type 1 units have about f1 links (e.g., about 3-10 links), type 1 units have about f2 links (e.g., about 20-30 links), and type 3 are fully connect. In one embodiment, there are more number of units (e.g., in H(1) layer) which have less number of connections to the lower layer units (e.g., in V layer), i.e., most units in H(1) have few connections to V layer units and few units in H(1) are fully connected to units in V layer.
Training with Samples of Varying Reliability:
In one embodiment, a measure of reliability of training samples may be provided with the sample data. In one embodiment, a default value for the reliability is assumed if not provided for a particular sample.
In one embodiment, an error function (to be minimized by training) defined over the training sample space (e.g., in a batch processing of an epoch) accounts for data sample reliability by including sample reliability factor as a weight in the contribution of the data sample to the batch error function, e.g., in the summation of the errors contributed from individual data samples.
In one embodiment, for example, a stochastic approach is used (instead of full epoch batch) to sample one (or several) training data sample(s) while optimizing the sample error function, and the sample error function is weighted by the reliability factor of the data sample. In one embodiment, the learning rate (e.g., the factor associated with the step to take in modifying the weights during the training) is modified based on the reliability weight for a given data sample used during the learning (e.g., in stochastic sampling of the data samples).
In one embodiment, some key data samples may be marked as the key representative samples. In one embodiment, an elevated weight is assigned to such samples during the training, e.g., to simulate the training with multiple instances of such training sample.
Preprocessing Prior to Classification and Training:
In one embodiment, one or more types of preprocessing is performed on the data (e.g., used for training or recognition) to focus on certain aspects of the data (e.g., image) in order to make the training and classification more efficient. In one embodiment, the preprocessing makes certain features to become more pronounced and easier to distinguish by the network (during and after training). For example, a filter such as Sabel filter is used in the preprocessing of an image to detect the line edges before feeding as training set for an RBM. In one embodiment, the preprocessing reduces features that may be less relevant in detection of pose and greatly simplify an initial step of choosing a more relevant expert system to further classify the image. In one embodiment, the preprocessing may actually introduce artifacts into the preprocessed image, e.g., a shadow on a face, may result in an edge across the face after an edge detector filter. In one embodiment, as for example depicted in
In one embodiment, the reliability of an expert module is trained by correlating the features indicating the domain of the expert module with the error encountered by the expert module.
Fuzzy Valued Feature/Label Output:
In one embodiment, multiple units are used to present various labels corresponding to a class of object. In one embodiment, feature detection system is used to train document classification based on learned (e.g., unsupervised) features corresponding to documents based on terms contained in the document (such as statistics of several hundred or several thousand common words), in one embodiment, latent semantic analysis (LSA) is used to provide the correlation between the terms or documents based on document-term matrix, and decomposition using orthogonal matrices and a low dimensional diagonal matrix (to a low dimensional space), e.g., by using single value decomposition technique (SVD). In one embodiment, RBMs are used for learning features, e.g., by limiting to top hidden layer to low number of units (dimensions/features). In one embodiment, the similarity between documents is determined by comparing (e.g., by cosine similarity) of their features. In one embodiment, the features (e.g., continuous valued) are correlated/associated with provided labels/classification (e.g., in supervised training). For example, in one embodiment, the labels indicate the type of document, such as legal, historical, fiction, scientific, business, manufacturing, technical, etc. In one embodiment, the layers are supplied to label units and correlation/association is learned via a correlation layer, e.g., by using an REM and using the features learned from unsupervised training. In one embodiment, more than one label may be provided during the training of a sample (e.g., a document). In one embodiment, the labels are binary (e.g., indicating whether the document is “technical” or not). In one embodiment, the labels are continuous valued (or multi-valued), e.g., having values in range [0, 1], to indicate the degree in which the document is classified by a label (or the membership function of the document in the label's class). In one embodiment, upon training the correlation/association, given an input data (e.g., a document), the reconstruction of labels (as output via, for example, correlation layer), presents the classification of the document based on those labels. In one embodiment, one or more labels are identified in the output, indicating that the document is determined to belong to both classes/types. In one embodiment, the output (multi-valued or continuous) indicates the degree in which the document is determined to be of the class/type. In one embodiment, the values output at the labels are thresholded (or further discretized) to simplify the presentation and further usage. For example, in one embodiment, an output less than 15% is zeroed, or an output more than 85% is turned to 100%.
In one embodiment, the membership function values presented by the output values in label units are consolidated to form a fuzzy number. For example, in one embodiment, the labels reconstructed from a handwriting recognition sample, show the membership function values in classes “1”, “7”, and “2”. In one embodiment, the labels for expression (e.g., facial) can be represented by fuzzy concept, e.g., smiling, laughing, sad, angry, scared, nervous, sleepy, apprehensive, surprised, and tired. And each label may have a degree of membership (e.g., degree of smiling compared to neutral) for a sample data, used for training. The labels may also be correlated based on the training data.
In one embodiment, various labels (i.e., their membership degrees) get correlated/associated with the features (e.g., at the top hidden layer of RBM or deep belief network) via training through a correlation layer.
Adding New Features:
In one embodiment, an already trained (e.g., unsupervised) feature detector (e.g., RBMs or a deep belief network) is used to provide additional feature(s). In one embodiment, one or more units are added at the top hidden layer. In one embodiment, the weights/biases related to the units already at the top (e.g., hidden) layer are fixed/frozen, and training (e.g., unsupervised) is performed to adjust the weights/biases related to the added units. In one embodiment, the added units represent a set of sub features that help enhance the reconstruction from top-down direction. In one embodiment, regularization techniques (e.g., limiting the weight amounts or weight decay techniques) or verification techniques (e.g., testing using reserved test datasets) are used to maintain or monitor generalization. In one embodiment, training samples with and without the features are provided to adjust the weights of the added units. In one embodiment, back propagation is used for fine tuning of the weights/biases. In one embodiment, the added units and the previous units are used to make association and/or correlation with labeled samples, e.g., during the supervised training.
In one embodiment, an expert classifier/detector is trained predominantly from one class to detect the distinguishing features of data within the class. In one embodiment, a set of feature nodes/units/neurons are added, e.g., to the top hidden layer of RBMs, for training to detect features of an additional class (for new expert). In one embodiment, the original expert classifier/detector is trained for a different class of objects (or expertise) to detect/determine the new features at the added units at the top layer. In one embodiment, the related weights/biases for the existing units at the top layer are fixed (i.e., prevented from changing) while training for the additional class. In one embodiment, redundant features (units) are eliminated (for example from the top layer) based on their strong correlation between those from the existing expert and the additional expert. In one embodiment, correlations between top redundant units are determined based on sampling from the data in the visible layer. In one embodiment, the correlation or covariance between top redundant units (e.g., at layer H(3)) are determined based on their biases and weights to the lower layer units.
In one embodiment, additional units are provided in lower hidden layers (e.g., layer H(2)) to allow more flexibility to handle more complex feature sets in a class of data/images. In one embodiment, the redundancy of such units are determined by strong correlation between the stochastic probability associated with such units, e.g., based on the value of logistic function of the total input to the unit. In one embodiment, in eliminating a unit due to redundancy, the weights linking the remaining unit to other units (e.g., in a higher layer) are updated by consolidating (e.g., algebraically) the corresponding weights from the existing and redundant units, in order to maintain the same total input to the top layer linked unit.
Focus of Interest with Variable Resolution:
In one embodiment, as for example depicted in
In one embodiment, as for example depicted in
In one embodiment, the resolution/size of segments varies in vertical or horizontal direction, as for example depicted in
In one embodiment, as for example depicted in
In one embodiment, features of an object (e.g., pose including rotation) is determined, and based on such features, features of sub-objects of other objects depicted in an image are extracted by preprocessing (e.g., mapping) a portion of an image into a segmented layout with variable resolution. Then, the mapped image (or portion thereof) is provided to a classifier or feature recognition system to determine the features from the mapped image. For example, in an embodiment, a frame depicted in an image is identified (e.g., a frame of a picture or a frame corresponding to a side of a building or a container). In one embodiment, based on the perspective/skew/projection of the frame other indicators), the image or a portion of image is mapped to a segmented layout for input to a network for further feature detection or classification.
In one embodiment, mapping of an image to a segment is done by averaging the intensity/color of the pixels falling into the segment. In one embodiment, summary information from the enclosed pixels of the image is attributed to the segment (e.g., texture, variance of intensity: color).
In one embodiment, a recognition/classification network or module (e.g., a deep belief network or RBMs) is trained using a variable segment layout associated with its visible/input layer. In one embodiment, an image is mapped to a variable segment layout before inputting to a recognition/classification network or module (e.g., for training or for recognition).
In one embodiment, an expert module uses/selects a variable segment layout to use based on other features of data/image determined by other recognition module. For example, a text recognition module may use a layout such as those, for example, depicted in
Estimating/Predicting/Localizing the Focuses of Interests:
In one embodiment, the locations of interest (e.g., the location of faces within an image) is determined by a scanning the image through a variable size window over an image at different location on the image, searching for example for particular features or signatures (e.g., head or face). In one embodiment, the locations of interest are determined, for example, by determining an enclosure (e.g., the smallest enclosure, such as rectangle or ellipse) around the object of interest, localize the object within an image. In one embodiment, the type of object (e.g., face) and its location (including orientation, skew, etc.) and other parameters (e.g., pose or identity of the object) are extracted and associated with the image. Then, such image and the associated information are used to train a feature detector/classifier to learn or predict the focuses of interest, by correlating/associating the image features with the locations of interest. In one embodiment, the image and various positions of interest are iteratively inputted to the system during training. The stochastic nature of the correlation layer, stochastically reconstruct parameters associated with the location of interest as output, e.g., using an RBM.
In one embodiment, a feature recognizer/classifier uses a data/image to extract features from an initial location (e.g., from the center of the image through a window or through a variable segment mapping). In one embodiment, based in the features determined, a set of one or more focuses of interest is reconstructed from the correlation layer (e.g., iteratively and stochastically). Then, the image is used to extract additional features from those predicted locations, e.g., through a window or a variable segment mapping. For each exploring location, a set of a location of focuses of interest are further predicted. In one embodiment, such lists of focuses of interest are consolidated and checked against the locations already covered. In one embodiment, the process stops after a certain number of locations of interest explored (for a given type of image), a certain number of features found, predicted location of interests were exhausted, certain amount of resources (egg., computing power expanded), or other rules.
Partial Image Training:
In one embodiment, partial images, e.g., masked or blocked, are used for training a detection/classifier module. In one embodiment, image samples are prepared by masking out the portions omitted e.g., by hiding the portion of image using straight edges through the image. In one embodiment, a randomizer generated masking parameters (e.g., the location of the mask edge). In one embodiment, the rendering module applies the mask to the image before inputting the masked image to the recognition module. In one embodiment, the masked regions of the image are filled with random fill color or random texture/pattern.
In one embodiment, as for example depicted in
In one embodiment, during the training, the weights/biases associated with unclamped V units are not allowed to change due to the learning step involving the training with the corresponding partial image. In one embodiment, the contributions to the error function related to the undamped visible units are ignored in the training step using the corresponding partial image/data.
In one embodiment, in the partial image training, the weight/bias adjustments for a learning step is modified by scaling the learning rate for a given unit (e.g., a hidden unit in H(1) layer) with the ratio of the number of its links traceable to the clamped visible units and the number of its links traceable to any visible unit. In one embodiment, similar adjustment to the learning rate is made with respect to a higher level hidden unit (e.g., in layer H(2)) by, for example, determining such ratio (indirectly) by tracing through layer H(1), or simply by estimating the ratio based on similar average ratio from the traceable units in H(1) layer. For higher hidden layers where each unit is quite likely traceable to every visible unit, the ratio is estimated as number of clamped visible units to number of visible units. In one embodiment, by tempering the learning rate, the impact of the partial image on the weights is tempered as well. In one embodiment, by limiting the adjustment of weights, the impact of learning from phantom or residual data/images from the unclamped is also reduced.
Learning Higher Details Iteratively:
In one embodiment, an effective (approximate) thumbnail is input to a visible layer of a feature detector/classifier (during training or search) by blocking/masking the data from the original image, from being clamped to the corresponding units in the visible layer, except as to sparse visible units, as for example depicted in
In one embodiment, the preprocessed thumbnail is applied to the visible layer, as for example depicted in
In one embodiment, the thumbnail pixel value is applied to multiple visible units, as for example depicted in
In one embodiment, the learning of features is initially performed by using thumbnails. In one embodiment, e.g., as shown in
In one embodiment, the stepwise learning of features from high level to more detailed takes advantage of the training weights established in earlier rounds of training.
Context Relationships:
In one embodiment, the datasets (e.g., images) include (or associated with) various objects or concepts (e.g., face, body, book, computer, chair, car, plane, road, and building). In one embodiment, classifiers are trained to detect high level signatures/features of various objects/concepts, e.g., by training the classifiers with (labeled) training data sets, including those with and without object features. Some data sets may include multiple objects or concepts, and therefore, the occurrences of the objects/concepts overlap. In one embodiment, a classifier may classify multiple objects/concepts. In one embodiment, the correlations between the objects/concepts are determined as the result of classification of various datasets. In one embodiment, a data-concept matrix is setup based on the classification of the data sets, and further analyzed, for example, by decomposition using orthogonal matrices and a (e.g., low dimensional) diagonal matrix (e.g., to a low dimensional space), e.g., by using single value decomposition technique (SVD). In one embodiment, this dimensional space represents various contexts (e.g., family, sitting, coworkers, house, office, city, outdoor, and landscape) that support or relate to various object/concepts. In one embodiment, each context represents/contributes a set of weights representing the relationships between object/concepts.
In one embodiment, upon detection or classification of a feature of an object/concept in a data/image, the distance of the data to one or more clusters representing various contexts is determined. In one embodiment, the clusters (or contexts) that support the data are determined. In one embodiment, a set of other concepts/objects are identified based on the correlation with the classified object/concept from the image. In one embodiment, the image/data is further explored, e.g., by other classifiers or feature/object detectors), selected based on the set of predicted/suggested concepts/objects. For example, in one embodiment, a face and a computer is detected in an image. Then, it is determined that such a data is consistent with several contexts (e.g., office and home) ranked in order of distance or consistency level to such clusters, or it is determined that such data is correlated to other objects such as keyboard, table, screen, room, etc., with various correlation strengths. In one embodiment, the classifiers or expert modules tuned for such objects are used to further detect the presence of such objects in the data.
In one embodiment, the contextual relationship between objects/concepts is used to further detect other objects/concept in data/image, by prioritizing and selecting the corresponding feature detectors/classifiers, as for example depicted in
Object Detection In Layers:
In one embodiment, an object/feature detector/classifier detects an object in a data/image, in one embodiment, the detected object may be part of or component of another object or detected for example based on the recognition of a partial image. In one embodiment, the structure of the object (e.g., the periphery, blob, coverage projection, or support regions) is determined based on localization of the object within the image (e.g., through reconstruction). In one embodiment, the potential objects/concepts in the image are determined, e.g., based on the context of the image or correlation with the context(s) of the detected object. In one embodiment, the visible structure of the object is removed from the image, e.g., as part of the objects in the image foreground. In one embodiment, e.g., with RBMs or deep belief networks, partial clamping of the input (visible) data is done for regions in the image not removed. Based on the context or correlation with other types of objects, corresponding detectors, e.g., RBMs or deep belief networks, are used to detect objects (which may be partially visible). In one embodiment, through reconstruction at the visible layer, the hidden/blocked portion of such objects is estimated/predicted. For example, this facilitates reconstructing background (if learned) or the rest of the face of a person (if learned). This approach can be executed continuously or iteratively to gather correlated collections of objects or their degree of possibilities based on the reliability factors. In one embodiment, more specific context may be derived based on each correlated (and for example expanding) collection of objects, and further information or proposition may be inferred (with a reliability factor) based on the image, by feeding the relationships and the reliability factors in a knowledge web.
In one embodiment, face recognition is performed on a partially blocked face in an image using a feature detector/classifier and database of known signature (vectors) associated with identified faces. In one embodiment, the comparison of detected features provides a matching probability measure between the partial image and a subset of those known in the database. In one embodiment, the reconstructed image at, for example, unclamped visible units representing the blocked portion, may provide full a face candidate for comparison with those images in the database.
In one embodiment, the consistency/reliability of a potential match with images/features (whether known or not) in a database is associated with the measure of uniqueness among the matches. The uniqueness measure indicates how uniquely the input image/feature is matched among the other images in the database.
In one embodiment, based on the context, there is a correlation between the hidden/blocked objects and the blocking object. For example, a dark glass covering a person's eye region has a strong correlation with the eyes. By training with the similar data/images with and without glasses, the correlation between the features of two data samples, given other common features may be established. In one embodiment, a data/image is searched by a trained feature detector/classifier. The features/labels indicating “wearing dark glasses” are activated based on previous training/correlation/association. In one embodiment, the region of dark glasses is identified (having correlation with the feature/label). In one embodiment, the value of feature/label is modified (e.g., forced off) by clamping the feature/label to the forced value. In one embodiment, such change/modification related to a feature/label is sent to a correlator/analyzer to determine the region on the data/image affected by the change, e.g., by reconstruction mechanism and comparison. In one embodiment, a threshold is used to limit the affected region on the image/data, e.g., based on relative changes in color, contrast, or intensity, size of region/sub-region. In one embodiment, the visible units corresponding to the thresholded region on the image/data are undamped, while the other regions are kept clamped to the corresponding image/data pixel/portion. In one embodiment, a reconstruction in V layer, based on the forced value of the label/feature is used to recover one or more candidates for the blocked image within the unclamped visible units. In one embodiment, a two step process to uncover the blocked images, uses the rest of the (unblocked) image as prior condition in predicting a likely covered image, as opposed to a top-down pass to reconstruct the whole image which may create more variety in the visible layer. In one embodiment, multiple pass from bottom up (inference) and top-down (reconstruction) is performed to obtain more candidate or likely candidates under the constraint of clamping the visible layer to the unblocked portion of the image.
In one embodiment, the correlation between the blocked object (e.g., eyes) and the blocking object (e.g., dark glasses) is learned by sequential learning or by using two instances of the classifier/feature detector (denoted as “C/FD”), as for example depicted in
Measure of Scale In Context:
In one embodiment, upon recognizing an object by a feature/object detector/classifier, the size of the detected object relative within the image and the relevant context(s) are used to correlate to size of other objects potentially in the image. In one embodiment, such estimates for the sizes of other Objects are used to locate potential areas in the image and the sizes to search for such potential objects. In one embodiment, this approach facilitates discovery of other objects in the image more efficiently given the location and size of window for searching for such objects have higher probability to yield detection of such objects.
In one embodiment, one or more pose detection modules (e.g., based on edge detection or color region/shape) are used to determine the pose of a face within an image/data. The scaling determined from the pose detection(s) is used to make more efficient detailed feature detection, for example, by scaling the portion of image containing the pose based on the size in a preprocessing step prior to inputting the preprocessed image to an expert feature detector.
Variable Field of Focus with Limited Data/pixel Points:
In one embodiment, as for example depicted in
In one embodiment, a quick scan recognition approach is used, based on resolution level of the focus widow. In one embodiment, at a given resolution level (e.g., R1 or R2, as for example depicted in
In one embodiment, the sizes of the windows for a given resolution are the same (e.g., for a given context). In one embodiment, the sizes of the windows for a given resolution are different depending on the location within the image (e.g., based on the context).
In one embodiment, the location of the windows are picked, determined, or adjusted based on the location of the object(s) detected in another windows, the context, the proximity and relative positions of the objects and/or the scale/size of the objects.
Learning High Level Features By Limiting Learning Space:
As mentioned in this specification, one approach to learn the high level (e.g., class of object such as presence of face, as opposed to for example the identity of the person based on detailed detection of facial features) is to detect the object/class of object based on a thumbnail (e.g., via preprocessing) of the data/image. In one embodiment, the training for a high level feature detection focuses on the structure of the neurons or units used in a classifier/feature detector. In one embodiment, the resulting feature units at top layer are limited to few features, while the training is used with data/images that may include thumbnail and high resolution data/images, including those with and without the targeted features. In one embodiment, a correlation layer is used to established the features correlation with labels by feeding the labels (e.g., via a label layer) to a correlation layer, or use a supervised training to train a classifier based on the labeled samples (e.g., using SVM).
Learning Via Partially Labeled or Mixed Labeled Training Set:
In one embodiment, the labels for supervised training or for making association with object features (e.g., already trained in RBMs or deep belief networks), may not reflect all the applicable properties of the sample training set. For example, a data/image containing a person and a chair may only be labeled as person. In one embodiment, as for example shown in
In one embodiment, specific annotations that repeat often (e.g., “Eiffel Tower”) (e.g., in a collection of images/data or a series of related images/data or within a large collection of data/images from various sources and various reliability) is taken as label for training association by adding an additional label unit (e.g., a binary unit) representing the added label.
In one embodiment, meta data such as the GPS data (or for example other accompanying metadata captured with images taken from mobile devices such as smart phones) are used as labels (e.g., continuous valued). In one embodiment, as for example depicted in
Search and Indexing:
In one embodiment, map reduce technique is used to handle “Big Data” processing across distributed file system and systems. The task, such as distributed search (among the machines) use small portion of the data (e.g., one chunk at the time) and provide the result to a central machine(s) for collection. An instance of search taskjob keeps the information about the search and identifies the result accordingly, so the result may be available or extended time. The result may get updated and available for use in real time.
Facial Expressions and Emotions:
In one embodiment, the weights on features that are affected largely by various emotional states or ranges are reduced in an attempt to distinguish the invariant features that would help identify an individual among a database of individuals associated with a set of features (e.g., invariant features). However, in one embodiment, the reduction of weight on the affected features will also impact (reduce) the distinctive features between individual labels.
In one embodiment, the expressions and emotional states are learned as features captured in the images. For example, in one embodiment, RBMs or deep belief networks regenerate or construct unseen images with new emotions, by setting the correlated label (for an emotion/expression) and letting the reconstruction provide an image in a visible layer.
Time Series and Video:
In one embodiment, multiple images are compared together to detect or infer transformation, e.g., translation, rotation, scaling of objects or features between the images. In one embodiment, the frames (images) from a time series collection (e.g., a video segment) is used to extract different poses of an object (e.g., a person's head), different expressions (emotions). In one embodiment, speaker recognition module, based on the analysis of sound track of audio/video tracks, identifies/distinguishes speakers and associates those entities to time segments in the audio/video tracks. An image extractor module uses the time segments to extract potential images at different poses of that speaker from the video track (in synch with audio).
In one embodiment, the feature detector is trained on various poses and expressions with many unlabeled samples before training with labeled samples to make association of features with labels (e.g., pose parameters, expression parameters, emotion states/degrees)
In one embodiment, the image transformation is modeled via a higher order Boltzmann machine, which links more than two units via a weight. A factored higher order Boltzmann machine reduces the complexity or the number of parameters (compared to non-factored version), where the weight (e.g., between 3 units i, j, and k) is factored into 3 mutual weights corresponding to each pair of units, in a multiplicative way: (wi,j·wj,k·wk,i·si·sj·sk), as schematically shown in
In one embodiment, as for example depicted in
In one embodiment, as for example depicted in
In one embodiment, the features recognized from previous analysis of the older frames are used with a predictive model such as Kalman filter to estimate the localization of the features in the current or upcoming frames. In one embodiment, for example based on such estimates/prediction, the analysis of frame (e,g., the current frame) is initially limited to a portion of the frame containing the estimated localization of the tracked features. In one embodiment, an area of focus of interest is used to analyze the portion of the data/image.
In one embodiment, stochastic sampling at hidden layers (e.g., based on an initial condition in visible layer) and the reconstruction in the visible layer reconstructs learned motion (in sequence) based on the learned weights, including autoregressive weights. In one embodiment, the learned (features) of the motion is correlated with a label via a correlation layer or other classifiers. In one embodiment, using one or more labels, the motion is reconstructed in sequence in visible layer via top-down regeneration. In one embodiment, a mix of motions are reconstructed based on a combination of labels (e.g., with varying degree).
In one embodiment, Long-short-term-memory (LSTM) which a recurrent type neural network is used to model the data in time series. In one embodiment, LSTM block includes sigmoid units (e.g., based on logistic function) to allow access to the block and control its functions (e.g., input, memorize, forget, and recall). It also uses product type units (with no weight) and summation units to direct the data through the block. In one embodiment, an LSTM module is trained via back propagation through time with iterative gradient decent algorithm.
Classifier and Complexities:
In one embodiment, linear models, such as perceptron, linear regression, and/or logistic regression are used. For example, perceptron is used for classification, e.g., in or out, accept or deny, and is trained by perceptron learning algorithm including a pocket version. The linear regression is for example used to determine (continuous valued or real valued) amount, based on squared error function and pseudo-inverse algorithm. The logistic regression is used for example in determining probability, based on cross entropy error, using a gradient decent algorithm. Noise and error in input data makes the nature of the training data probabilistic. The VC (Vapnik-Chervonenkis) dimension for a Hypothesis set (i.e., the most points that can be shuttered by the hypothesis set) is related to hypothesis set's growth function, and in one embodiment, the VC inequality (in terms of growth function and number of training samples) provides a rule of experience for the number of points needed for training (e.g., >10×VC dimension). The VC inequality places an upper bound on the probability of the out-of-sample error (i.e., the generalization error) is within the in-sample error by a given error margin and a given number of in-sample (training) data. In one embodiment, a maximizing likelihood approach is used to select a hypothesis from the hypothesis set that results in maximum likelihood of getting the data given the hypothesis. In one embodiment, the learning with logistic regression uses a cross-entropy error log(1+exp(−ynWTxn)) with (xn, yn) representing the labeled data point and W is the weight matrix to be optimized. In one embodiment, the optimization uses a gradient decent approach by using variable size step (large to small). In one embodiment, the step size is proportional to the gradient which fixes learning rate (appealing as a multiplier for the learning step). One embodiment uses an adaptive learning rate. In one embodiment, the default learning rate is 0.1. In one embodiment, the number of iterations of epoch is limited to a maximum (early stopping), in order to avoid over fitting the noise-'error and deteriorate generalization by increasing the out of sample error. In one embodiment, in order to tackle the problem of local minimum, the optimization starts at different initial values of weights (e.g., based on heuristic). In one embodiment, the “temperature” is used to escape local minimum, e.g., in RBM learning, the optimization starts at a high temperature, to allow escaping the local minimum. In one embodiment, a stochastic gradient decent is used by taking one data sample at the time, resulting in generally a simple, cheap, and random approach to optimization in comparison to batch optimization where all data sets are used in each step of optimization. In one embodiment, a binary search method is used to explore along the direction of error gradient. In one embodiment, a conjugate gradient is used to estimate the second order error from previous data points. In one embodiment, a multiclass classification is approached based on binary decision, i.e., 1 vs. all, 2 from the rest, etc. In one embodiment, non-linear transformation is used to optimize based on a feature in a transformed space. In one embodiment, the VC dimension of the neural network is approximately the number of weights and biases. In one embodiment, a regularization approach is used to kill some weights (e.g., in random) to enhance generalization (and reduce over fitting). In one embodiment, a genetic optimization is approach is used. In one embodiment, a regularization approach is used to limit the choice and ranges. In one embodiment, a validation is used to test the generalization, by dividing the sample data for fitting and cross comparing the error. In one embodiment, kernel methods are used for small labeled data and top features to model the classification. For example, one embodiment uses thousands of unlabeled training set with various orientations to learn features (including the orientation), and it uses few hundred labeled training sets to discriminate orientation (with regression for angle). In RBM training, the number of training cases may be less than number of weights as long as the number of pixels is much more than weights, because there are a lot of features that can be extracted from pixels. In one embodiment, the discriminative training (e.g., based on labels) quickly fits the data, and it is stopped quickly to avoid over fitting. In one embodiment, a weight decay technique is used to implement regularization in learning. In one embodiment, about 20% of the data samples are reserved for validation (and not training). In one embodiment, cross validation is used to conserve the number of data sample for fitting. In one embodiment, the probabilities indicating the certainty of inferences based on the correlated training sets are tracked, for example, to infer one pose from a different pose.
Feature Extraction:
In one embodiment, we note that people of same ethnicity or region (or members of the same species of animals) generally recognize each other better. For example, all tigers look the same to an average human, but tigers themselves can recognize each other very easily and efficiently. Or, a Middle Eastern person can distinguish other Middle Eastern people more accurately and efficiently, than what a Chinese person can, among the same set of people from the Middle Eastern origin (or the same set of pictures of people from the Middle Eastern origin), assuming that the Chinese person never lived among Middle Eastern people or lived in that region of the world.
The same is also true (for example) for the case of the identical triplets in humans, which can be distinguished easier by themselves. In addition, their parents can distinguish them easier than the rest of the general public can. The reason is that an average human can see a tiger as an animal with 4 legs and stripes, similar to a big domesticated cat, as the dominant features, however, the tigers see or focus on some other features or more details of stripes, to distinguish themselves or as they see themselves. Since a tiger's eyes and brain are trained by looking at a lot of other tigers for many years, their distinguishing features are all set accordingly, to focus and look for the right features or sub-features, to make the distinction. For example, they may look at the ratio of 2 lengths on 2 specific stripes, or width of a stripe near the nose, as the focused or re-focused features, to find or classify or distinguish the other tigers or objects or subjects. Such specific features may be transparent to the human eye and brain, as they do not know what to look for in a huge volume of information received by the eye or brain. It is also consistent with the fact that a zoo keeper (dealing with tigers for years) can distinguish them much easier than an average human, as the zoo keeper has seen many tigers, and thus, her brain and eyes are trained for those features and characteristics.
So, sub-sampling the input from all universe (for humans, for example), or UH, is very critical for training purposes, to train appropriately, for a given task. Filtering or focusing or zooming in a few features (FF), out of, e.g., billions of patterns or features available (FU), on the sensory basis or recorded or obtained, when selected appropriately for the task at hand (TA), reduces the training time and cost, and increases efficiency and accuracy of recognition and classification and appropriate response. Mathematically, we have:
TA→FF
Wherein FF⊂FU
Wherein UH→FU
Wherein “arrow” symbol indicates that the right hand side item is obtained from the left side item.
Large or Voluminous Input Data:
The same is true for an infant (e.g., 5-month old, at the early age) listening to all the noise and voices around herself, e.g., possibly spoken in English and French by bilingual parents or nanny, and the noise from highway outside in the background, as well as the very noisy and loud fan on the ceiling, or the voice of her grandfather and grandmother, with the TV announcer or anchor in the background shouting about a recent news or an advertisem*nt about a car. She receives a large amount of voice and noise data by her ears (or internally from vibration on the ground or floor), but in the first few months, she gets all of the data with the same weight and importance. Overwhelmed by the large incoming data, she mostly ignores most of the input data, even the voices of her parents, that are supposed to be important to her well-being. After a while, though, she will understand that her parents' voice are more important than the noise of the cars outside or fan on the ceiling, even if they are very loud or louder. So, she will tune and filter or put more weights for those features or voices, as she gets trained on distinguishing between the voice, noise, music, warnings, background noise, dangerous signs or screech/scream, or angry tones. The same is true for vocabulary and grammar in a language.
It is the readjusting, reassigning, or rearranging the orders or weights or features, which focuses or re-focuses the learning subject on new or different features at the different stages of learning process, including distinguishing features or pattern recognitions. Thus, the learning process is dynamic and self-adjusting or adjusted by a trigger or test against a threshold or from an outside input. It evolves, as it gets more sophisticated, for more capabilities.
For example, in one embodiment, as the time passes, a subset of input features (F1 ( . . . )) are selected at a given time (tN), compared to the previous time (tN−1), until the subset becomes the same as the set itself from the previous time. Then, there is no need to sub-select anymore, to reduce the set of features. Thus, the optimization process stops at that point, and the final sub-set is selected and obtained. Mathematically, we have:
F1 (tN)⊂F1 (tN−1)
For ∀ti
Until we have: F1 (tM)=F1 (tM−1)
In machine learning, in one embodiment, we initially teach the machine the language or other things or subjects without any rule or grammar, just by training samples, and usually by sheer number of training samples. Then, on the second phase, we teach or input the machine some basic rules, e.g., Fuzzy rules or rules engine. Then, on the 3rd phase, we train the machine with more training samples, simultaneous with more rules being input, to have some order to the training and samples, which is a very powerful way of learning rules and getting trained very efficiently.
In one embodiment, the machine learns one language (or concept) by training samples only. Then, we teach the 2nd or 3rd language by mapping and templates, based on the first language, especially with grammar or rules, for more efficient learning.
In one embodiment, the machine learns the OCR or recognition of the text based on strokes or basic curves which in combination make up the letters. For example, for letter “t”, we have the shape “1” plus the shape “−”, with their relative positions with respect to each other. So, we have 2 basic strokes in our dictionary, so far, namely, “1” and “−”. Once we do this for all letters and marks on keyboard or in language, we get a lot of basic strokes in common in our dictionary, which we can re-use for others.
In one embodiment, the machine learns based upon the strokes, first. Then, it learns based on the shape of letters, e.g., “t”. Then, it learns based on the words, e.g., “tall”. Then, it learns based on the phrases, e.g., “tall building”. So, in multiple stages, it graduates from basics to more complex structures, and reads phrase by phrase to understand the text, similar to an experienced human speed reader, who can read the articles in a specific subject of her specialty very fast, faster than an average person, in which she scans, reads, and understands the text by chunks bigger than simple letters or words.
In one embodiment, instead of the machine learning in different stages, the 1st machine learns the strokes, and feeds to the 2nd machine, which learns the letters, and feeds to the 3rd machine, which learns the words, and feeds to the 4th machine, which learns the phrases, and so on.
In one embodiment, we have a neural network, with multiple hidden layers, each layer representing a more complex structure, for learning process, e.g., the first one for strokes, the second one for letters, the third one for words, the fourth one for phrases, and so on. In one embodiment, we have enough processing elements (PE) in each hidden layer for our learning machine, with artificial neural network (ANN) structure, so that it can accommodate a language with all its strokes, letters, words, and phrases. For example, for English language, for the second layer, we have 26 PEs, if we only deal with the 26 English letters of alphabet, and only with lower case, and only in one type and format, as our universe of possibilities, for recognition purposes. In one embodiment, with enough training and samples, with all the weights and PEs set, we set all the recognition for letter-level recognition in the language at the second hidden layer.
Data Segmentation or Classification, for Data Processing or Recognition:
In another word, in our daily life, we routinely receive a large amount of data, in which a first subset of that data may be useful for the first task, and a second subset of that data is useful for the second task (analysis, recognition, or distinction). So, for the first task, if we get rid of the rest of the original data that is not used, as useless data, to save storage space or increase recovery or retrieval speed, then, later, for the second task, we do not have the proper data for the second subset of the data, to complete the second task properly. However, if we have enough space to keep all or most of the original data, at least temporarily, to keep most or all of the second subset, or to keep all of the original data intact, then our second task can be accomplished successfully or relatively more successfully. One example is when we get voice data and image data from the same source at the same time, and the first task is to understand the voice data, and the second task is to understand the image data, which (in turn) comprises text images and pictures, which corresponds to subtask of text recognition (e.g., OCR) and subtask of picture recognition (e.g., face recognition)
Data Segmentation or Classification, for Separate Optimization:
Another situation is when, for example, we have a compound image, which includes the combination of thin sharp line drawings and color photos. If one tries to optimize or improve the quality of one region or one type, e.g., the color photos, the other regions or types can be defocused or smudged, e.g., un-sharpening the lines, which destroys the crisp and sharpness or continuity of the thin line drawings, effectively destroying the black/white line drawings (or the text part) of the image. That is, we degrade the second part of the image, by improving the first part or section or type or region of the image.
Thus, we may want to classify and segment the original data, so that each part or section or type is optimized or improved separately, as the optimization is not universal for all parts of the image or data. For example, to read the text better,/improve the text quality, the rest of the image (e.g., the color pictures) may get degraded at the same time. Thus, in one example, it is better to segment and classify first, and then do the optimization per region or type, or per task, as needed, e.g., to optimize the text only, and leave the rest of the image intact.
Separate optimizations with different degrees of optimization or filtering or transformation can be applied to different regions of an image, as shown in an example in
Optimization:
Note that the optimization is not universal. For example, we take a picture at night with a camera using a flash light, from a metal surface, with high reflection of light, which overwhelms the resulting image, with a big blob of high intensity light reflected and captured in the image, and the text underneath is not visible at all, at the first glance. However, if one plays with and adjusts contrast and intensity/other image parameters, at one point the text on the image from the metal surface becomes visible, of course, at the expense of the rest of the image (as it becomes degraded). That is, the optimization is usually localized and for specific purpose. That is, generally, the optimization is not universal, or not for all-purposes, or not for all types of data, or not for all regions of image.
Scope of Search:
In one embodiment, we start from task or goal, to limit or set the scope of search or result or filtering. Thus, the question (or task or goal or what we are asked for or looking for) ultimately determines how to optimize (or view or filter or twist or modify or convert or transform) the data or image. That is, the assigned task gives the proper context or scope, so that we can focus to search or filter or optimize for the corresponding answer or result. That is, there is no single answer or filtering for all tasks or goals. Each task demands and corresponds to its own filter or transformation or result. That is, the scope of the transformation or filtering is determined or limited by the task (or assigned goal at hand), itself. Another way to look at it is that once we define the “noise”, as what the noise is in that context or environment, then we can define the filter that reduces that noise, which sets the goals or tasks for our optimization.
Relationship Possibilities:
Note that there are extremely large amount of relationship possibilities from a very limited finite set of data. For example, let's look at the tiger example again. The tigers may have only about 20 different stripes, as an example, as a finite and very limited set of data, e.g., expressed by a set of pixel data representing an image, with specific 256 color resolutions for each component of color RGB data and at 600×600 dpi pixel resolution in 2-D orthogonal axes/directions.
However, we can define much bigger number of relationships (e.g., hundreds of billions (although some are not independent of others, and can be derived from others)) between these 20 stripes, e.g., ratio between width and length of each stripe or between stripes, or angles or curvatures of each stripe or multiple stripes, as different combinations of ratios of these features, which by far dwarfs the number or size of the original data corresponding to 20 different stripes. However, from among all these ratios (e.g., billions), maybe, there are only a handful, say e.g., 3 stripes behind the nose and ear for each tiger, with their corresponding lengths or their ratios, that produce only 6 numbers, as an example, that are the determining parameters to distinguish any tiger in the set. So, only 6 numbers are needed for our recognition process. However, this is not readily apparent, when we look at the original 600×600 image, or when we look at the billions of calculated ratios or relationships or lengths.
Thus, one has to know what to look for, which is usually a subset of the original data or relationships or features, to make the recognition in the universe of the objects, to recognize all or most of the members of that universe. To zoom and focus on those 6 numbers (as an example), one can reduce the calculation and memory needed to do the task of the recognition, the same way a tiger recognizing her own family and siblings in a fast and efficient way, with much less analysis than an average human would do, to recognize the same tigers, if it is possible at all.
In one embodiment, we do not know what we are looking for, e.g., in a “big data” analytics. In those situations, we guess at some pattern or feature, as candidate(s), from our history or experience or library or other user's experience or using expert's opinion in other or similar situations, to test the hypothesis, to see if the first candidate yields anything in search or recognition. If not, then the system moves to the second candidate, and so on, to exhaust the list of candidates. If any candidate yields anything, then we continue the analysis on that candidate and follow that pattern or feature. In principal, we may end up using multiple candidates and find all the corresponding patterns or features.
The candidates can also be picked up randomly from our universe of possibilities or library, in some situations, e.g., where there is no preference in mind, or no experience or history on file. For example, for images received, we may want to see if we can find any face in the images, or find any text, or car, or any function with sinusoidal behavior (e.g., periodic), e.g., with intensity of pixels going up and down periodically, e.g., in a bar-code image with parallel stripes with some periodicity (T) or frequency (f).
Multiple Recognizers or Classifiers:
Let's look at the human/face recognizer engine or module or software. If a recognizer is trained for pictures or images of people from Middle East to distinguish among them (first module), and another one is trained from Chinese or oriental people or training samples or images (second module), then we do not want to re-train one module to change its weights, e.g., neural network weights, to convert or optimize first module to become second module. Thus, we want to keep both as-is, as each module is optimized on a subset of samples. So, in a first level, we figure out and sort based on the regions of the world, on a coarse and quick basis, and on the second level of analysis, we send the result(s) or images for analysis to the first module and the second module, and so on, which are optimized based on a subset or region of the world or population, to do an accurate recognition in that subset, only. This hierarchical model can expand to many layers, to go to subsets of a subset, for further analysis. So, in this embodiment, to be efficient, not all recognitions are done in one level or one recognizer or one neural network, as an example. See e.g.,
In one embodiment, we use our search engine as multiple expert systems, e.g., it has a section for English language, a section for French language, a section for car engine, a section for food and wine, and the like. See e.g.,
In one embodiment, the context is hard to guess from one sentence (Stext). So, we usually need a large sampling or history or third entity input. However, in one example, Z-web itself can also help set the context right. For example, if we have 2 context candidates in mind to try, namely Context-1 and Context-2, then the corresponding Z-webs can be examined, namely Z-web-1 and Z-web-2, respectively. Then, if e.g., we have more nodes (or higher weights or reliability factors) related to our Stext from Z-web-1 than that of Z-web-2, then we can conclude that between the two contexts, Z-web-1 or Context-1 is probably a more relevant context. So, between the 2 choices, we choose Context-1 as our context. See e.g.,
In one embodiment, we have multiple recognizers or classifiers, with different degrees of complexity (and overhead and cost and accuracy and depth analysis and focus). We cascade or chain them as from simple to more complex ones in series, one feeding the other, so that if the answer is NO for one level, it does not have to try the next level classifier any more, and stops the process at that point, with exit from the loop. If the answer is YES, then it tries the next level classifier, which goes more in depth, to find more about the object, or classify more specifically, based on the result of the previous classifier (which had a broader scope of inspection). For example, first, we find it is a text document, then, we will find out it is a book, and then, we will find out it is a novel. Obviously, if it were not a “text” determination at the first level of classification, we did not have to activate or teed it into the “book classifier” or “novel classifier” in the next steps, as they would have been useless, as their expertise or focus would not be needed at all. Thus, the system is more efficient and more compartmentalized and more expert-oriented and more depth analysis and deeper classification or recognition, now.
To combine classifiers, in one embodiment, for classifiers which only return the selected class or rejection, we can use the following methods to combine the classifiers: maximizing posterior probabilities, voting method, or Dempster-Shafer theory. To combine classifiers, in one embodiment, for classifiers which return a ranked list of classes or categories, we can use the following methods to combine the classifiers: Borda counts or logistic regression method. To combine classifiers, in one embodiment, for classifiers which return a ranked list of classes or categories, together with the classifiers which return a measurement associated with each class, we can use the following methods to combine the classifiers: weighted sum, ruled based, fuzzy integral model for classifier fusion, associative switch, or trained perceptron. To combine classifiers, in one embodiment, for all classifiers of any type, we can use the hierarchical decision making method. To combine classifiers, in one embodiment, we use and add the complementary classifier, to improve the performance of the combination. To combine classifiers, in one embodiment, we use unanimous voting or majority voting scheme for combination.
Classifiers:
In one embodiment, we have the nearest neighbor rule for training samples and the closest prototype, for assigning the corresponding class, to optimize the classification. In one embodiment, we get a binarized image. Then, it is thinned to get the skeleton image. Then, the system extracts a set of features to classify (as a separate class for recognition).
In one embodiment, we use a Markov chain-based classifier, with state transition probability depending only on the current state. For example, for Markov chain, we can represent an object with its boundaries or border or edge line, which is represented by a collection of points connected together using short straight lines, which can be represented by a chain of points, going from one to next, based on a direction and distance values, to set or get to the next point. So, starting from point one, we can complete the loop and find the boundaries or border of an object, and each point depends on the prior point in the chain, which is based on Markov model.
In one embodiment, for classification, we use “Fuzzy c-Means Clustering Method”, with a fuzzy pseudopartition or fuzzy c-partition of our set (where c is the number of fuzzy classes in partition), in terms of cluster centers, and using inner product induced norm in our space (representing distances in that space). The performance metrics measures the weighted sum of distances between cluster centers and elements in those clusters. We want to minimize such a function. First, we choose an initial pseudopartition. Then, we calculate the c-cluster centers in the form of:
Si=(Σk[Pi(xk)]nxk)/[Pi(xk)]n)
for the initial pseudopartition and a specific n, wherein {P1, P2, . . . , Pc} represents a fuzzy pseudopartition, xk represents a set of given data, and Si represents the partition (with association being strong within clusters, but weak between clusters).
Then, we update the values, for (t+1) instance: If the distance measure ∥xk−Si(t)μ2>0, then we have:
pi(t+1)(xk)=(Σj∥xk−Si(t)∥2 )/(∥xk−Sj(t)∥2))(1/n−1))−1
wherein j runs from 1 to c. If ∥xk−Si(t)∥2=0, then we have: (ΣiPi(t+1) (xk)=1), for (i∈I). Otherwise, we have: (Pi(t+1) (xk)=0).
Then, we compare the values for instances t and (t+1). If the difference (or the distance in our space) is less than a predetermined threshold, then the system stops the process (and exits the loop). Otherwise, the system (or controller or processor) increases the counter t by 1, and repeats the loop again, as shown above (until it exits the loop at one point in the future).
In one embodiment, in the manipulation of Z-web, for any fuzzy clustering, we use the method detailed above, for clustering and recognition purposes.
In one embodiment, for pattern recognition or classification, we use clustering tree, e.g., with Euclidean distance or Hamming distance, or use Fuzzy Membership Roster Method. In one embodiment, for fuzzy pattern recognition, we use the degree of membership of an object to associate the object to a class or multiple classes (in contrast to the conventional or classical classification goal or method). That gives us more flexibility for classification. In one embodiment, we use a minimum threshold, for min. value for the membership, below which the membership is set to zero.
In one embodiment, we use fuzzy syntactic method for language(s) and its respective grammar, which governs the rules for string of symbols that makes up the language (or replaces the language or makes a template or encodes the language or summarizes the language). In one embodiment, we use fuzzy grammar, which is not crisp, and is based on overlap and partial relationship, with membership function or value expressing such a relationship, relaxing the strict requirement by crisp or conventional grammar, making it more compatible with natural language processing and human language. In one embodiment, we use multi-level hierarchical classification method, for class, subclass, and so on, at different levels.
Minimum Number of Objects Needed:
For the minimum number of objects needed for defining or describing a situation or relationship, let's look at one example. Let's assume a mother is teaching her new-born son how to speak English. If in that universe, there is no radio, TV, CD, or book available, and there is nobody else available to talk to them, then the distinction between “I” and “You” would be very hard for the son to understand, as he may think that “You” is his first name, at least at the beginning, because there is no third person to talk to, or other interactions with a third party, or a video to watch with a third person talking in it, that can set the meaning of “You” properly for the son. So, it would, at least initially, be very confusing for the son. So, for any given situation, one needs a minimum number of objects, or a “minimum” arrangement or setup, to define the situation properly and define the relationship between those objects properly.
Minimum Vocabulary Needed:
In addition, in a universe with 2 people only, there is no need to have a third person referral, e.g., “he”, “she”, “hire”, or “hers”, in the language, because there is no use for it at all, and it may not make any sense to have an extra baggage or term or name or reference in the language for a non-existence concept or object. So, in one embodiment, we can reduce and simplify the minimum vocabulary set needed to operate or communicate in that universe, by eliminating the excess baggage or words or terms.
Age Progression Model for Pictures:
For all human races, genders, and face shapes, one finds NP clusters, based on PD dimensions or number of parameters extracted from each sample. For each type of face, from NP possible types, one defines or designs a set of SA age progression templates for different ages of the person, which can be morphed in a series, as the person gets old. This can be done analytical or mathematical based on coordinates and curves defined for each face. This can also be done by using a series of pictures from a real person at different ages, to fit the model, or for learning using a neural network, or use as template for that type of face. Of course, the larger the number of examples or number of people, MP, the better the template will be for the fit. So, we need a lot of pictures of many people of different face shapes, at different ages. See e.g.,
To be efficient, we use a subset of those images, as the ones from similar ages do not help any further. For example, the image of a person between 41 and 42 years of age does not generally change much. So, it is not much helpful to store both. But, image of a person, for every 6 months, between the ages 2-3, changes a lot, and so, it should be stored more often/frequent at younger ages, as an example. So, in a series of age progression images, one can mark the major changes as the main ages or images to keep, as a good sampling set or collection. So, we define the difference between 2 images, e.g., at pixel level, as difference between pixels, divided by the value of the original pixel, as the normalized value, to compare the relative changes in images at different ages, to find the largest jumps and changes at different ages.
So, we can find the major changes from real images. Alternatively, we can find the major changes based on prior knowledge from many thousands of images of other people, to categorize the age brackets, and find the break points, or alternatively, from medical database, indicating the expected changes for an average human, for various major changes in shape, height, face, or features, e.g., beard or hair loss, versus age brackets or break points in time axis, e.g., for the people from Middle East, as a subset of population, with expected values.
Note that if the scales or angles of view of the faces are not the same, in 2 images, then, before comparison, we have to adjust or normalize to one size or direction, so that we can compare them meaningfully. To adjust the size, one measures e.g., the length or width or diagonal of the face or nose, as the calibration metrics or normalization factor, to scale one with respect to the second one. To adjust the angle, one has to look at the symmetry or direction of the nose with respect to the oval of the face or position of ears or eyes, as an example, to estimate the direction and angle of rotation, to adjust the direction of the face, or normalize the direction. For angle adjustment, we use models we have for rotation of the face as templates, to morph one to the other direction. The models are based on NP possible types of the head or face in our database, described above, to get a better result. After rotation of the face, we compare it to the second image, which has about the same orientation. Then, it is a fair comparison. In one embodiment, all faces are turned to the same direction for comparisons, e.g., front view, only. See e.g.,
In one embodiment, instead of rotating the image, we look for an image of the same person corresponding to the same rotated view or angle, from the same age category, if it is available in the database. Then, no rotation is needed, and less computation power is used.
To do the morphing from the first angle to the second angle, for the image of the face, we model the face or head as a mesh with contours, with points on it, as a template for each angle (or direction or view). Moving from one point from the first angle template to the next corresponding point on the second angle template is equivalent to moving the contours or meshes slightly around its current position. We choose the morphing in small increments for angles so that the differences are manageable by slight movements of contours or meshes. We can model the slight movements by vectors and translations and rotations of vectors, or a combination of them, in a series of instructions to morph properly, for piecewise regions of mesh or contour on the face.
Another way to do this vector modeling is by modeling and fitting a real picture or image of a person at different angles point by point (by coordinate in 3-D), then connecting the neighboring points to get contours, and from the series of contours get the mesh, modeling the face. Then, we have this repeated for different angles for the same person. Then, store the data for that person, representing one of the NP possible types, that corresponds to that type of head or face, in a database, for future referral and comparison.
During this process, for a given first point on the mesh, we find a corresponding second point on the template for a second angle or view. Then, on the coordinate of the 3-D model, with 3 components (x, y, z), we find how much the first point (x1, y1, z1) moved (to the second point (x2, y2, z2)), which is:
(x2-x1) in x-direction
(y2-y1) in y-direction
(z2-z1) in z-direction
We can repeat/get this calculated for multiple points, and then model all of those changes in coordinates in 3-D for those multiple points, using translation, scaling, and rotation, or a combination of the above. That would be our final recipe or series of instructions or steps for morphing process.
Please note that the translation is based on (x2-x1) in x-direction, (y2-y1) in y-direction, and (z2-z1) in z-direction. The scaling is based on (A x1) in x-direction, (B y1) in y-direction, and (C z1) in z-direction. The rotation is based on matrix of rotation, e.g., in 2-D expressed as a 2×2 matrix (M2×2), with the following 4 entries (Mij), for a clockwise rotation by angle a on a 2-D plane, as one example:
M11=cos α; M12=−sin α; M21=sin α; M22=cos α
In one embodiment, we use 3 types of templates for face model in 3-D (dimensional) for face recognition, or after scanning the face (with a light, scanner, or by a 2D image or multiple 2-D images), or for storage, library, or comparison, alone or in combination: (1) wire mesh using thousands of points on the face, (2) contours of face for topography and geometry, e.g., cheek bone curves and structure, and (3) semantic model, which models the face based on the general semantics and description of the face, e.g., “big nose” or “small lips”, which are Fuzzy descriptions, with corresponding library of descriptors and shapes, plus rules engine or database, defining those beforehand, so that we can store or reconstruct or combine Fuzzy features e.g., “big nose” and “small lips”, and e.g., make up a face from descriptors later, or compare 2 faces just using descriptors without reconstructing the faces at all, which is very fast and cheap, for a Fuzzy match or closeness degree. In one embodiment, we use many small steps between Fuzzy descriptors on the scale or axis, to have differentiation between objects more easily and have a good coverage for all samples in the defined set or universe, e.g., for “height” property, we will have: “short”, “very short”, “very very short”, “extremely short”, “unbelievably short”, and so on. See e.g.,
The method of recognition mentioned above is helpful as one of the parameters for face recognition, or validation for identity of a person, using pictures of different years or ages, to find a person, Identity recognition, in turn, is a factor for determination of the relationships between objects and humans (or other subjects), and to build such a web of relationships or Z-web from all these determinations, like a tree structure, with nodes and branches, with strength of relationship and reliability of the determination e.g., symbolized with the thickness and inverse length of the branches (respectively), connecting the concepts as nodes, for example, for display purposes, for visual examination by the user (which we call Z-web).
In one embodiment, we have a picture, or multiple pictures of a same person, possibly from different angles, and then we feed that to the system, and then from library, based on shape comparison (e.g., features and parameters of the head in N-dimensional feature space), the system chooses the most possible type of head, out of say e.g., 105 types it has, to suggest that as a model. Once we have the model, we fit those one or more pictures into that model, and construct point by point or mesh structure or contour map of the face. The model has some parameters as variables, which can be adjusted in 3D using those 2D images as input, which gives elasticity to the form of the face and head in the 3D format, for minor adjustments to the 3D model in computer (which can be displayed for the user, as well, as an option). In addition, the same 3D model can be input to a 3D printer, or 2D rendering image printer, or laser induced bubble printer (in plastic or glass), to construct the same head in the solid format, e.g., in glass or plastic or polymer.
In one embodiment, we have e.g., front view of a person, e.g., in a picture or image. Then, we use slanting or some deforming lens or filter or translational transform(s) to change the shape of the face slightly, and store them as the basis for the rotating or moving head slightly, from the front view position (from its original position, with small perturbation or movements), in the library. So, we can use them as eigenfaces for frontal or near frontal sideway faces, for the future face modeling, face replacement, face recognition, face storage, as linear combination of eigenfaces, face approximation, efficient storing of faces, coding the face, and comparison of faces. See e.g.,
In one embodiment, we have orthogonal or orthonormal eigenfaces as basis. In one embodiment, we have non-orthogonal or non-orthonormal eigenfaces as basis, e.g., some being as linear combination of others, which is less efficient for recognition (and being too redundant), but easier to generate the basis functions, due to less constraints on basis functions. In one embodiment, we obtain eigenfaces from thousands of samples, by cloudifying or fuzzifying or averaging pixels in large neighborhood regions for the samples, in the first step. Then, optionally, we can stop there, and use the result of the first step as our final answer, as eigenfaces. Or, we go one more step, in another embodiment, and we average the first step results together, to get even more “cloudy” images, as our final result, for our basis, for eigenfaces. Or, we go one more step, in a loop, recursively, in another embodiment, and we average the averages again, until it is cloudy enough or we reach N loop count, and we stop at that point, yielding our eigenfaces. Then, any given face is a linear combination of our eigenfaces. See e.g.,
To remove redundant eigenfaces from our basis functions, e.g., to have an orthogonal set, we try or choose one eigenface, and if we can write it in terms of linear combination of others, then that chosen eigenface is redundant (and not needed) and can be removed from the set. In one embodiment, to make some image fuzzified, we can use fuzzy parameters, rather than crisp ones, or use dirty or oily lens for image, or use defocused lens or out-of-focus lens for images, as a filter or transformation or operator, to get the cloudy or average effect between pixels.
In one embodiment, for face recognition, or eyes or any other object, we have Sobel operator or filter or matrix or convolution, based on gradient or derivative, so that the operator finds the gradient of the image intensity at each pixel, e.g., the direction of the largest increase for pixel intensity (with the rate) or contrast, as an indication of abruptness of changes in the image, to find the edges or boundaries, to find the objects or recognize them. In one embodiment, other filter kernels, e.g., Scharr operators, can be used for edge detection or gradient analysis.
In one embodiment, for face recognition, we use edge detection or other object recognition methods to find eyes (or nose), first, as an anchor point or feature. Then, from the eyes' positions, we know relatively where other parts may be located, if it is a real face, based on expected values or distances based on face models in library, e.g., as a probability distribution or expected value or average value or median value, for distances. See e.g.,
Z-Web Representation and Manipulation:
The graphic representation of Z-web makes it easier to visually understand the strength of relationship and reliability factor, among other factors embedded in the Z-web, as explained in other parts of the current specification. The graphical representation also mirrors fuzzy parameters, as the human visual perception is not crisp, but it is fuzzy, similar to natural language processing and expression.
To get an object, one searches for nodes on the Z-web (e.g., using an index on a database or listing, using a query), and once the node is determined or found, the connectors and branches coming to or from that node are examined for determination of the reliability and other factors mentioned in this disclosure, from the numbers or sizes or dimensions associated with the connectors and branches, e.g., the thickness or length of the branch between 2 nodes. The “circle of influence” is based on (in different embodiments): the neighboring nodes, or N-th neighboring nodes, or nodes within radius Rnode, centered at that original node, as a hyper-sphere, in the m-dimensional Z-web space, with m coordinates. The circle of influence gives us the guidance as to where and how far we should go for related nodes or concepts or objects, in the branches, to find other objects or recognize objects or find the reliabilities or confirm the objects. Sometimes, the influence of the circle of influence dies off gradually, and not abruptly, using a fuzzy parameter to model that behavior. In other embodiments, the influence of the circle of influence dies off abruptly, which is an easier model to handle and calculate for.
The user interface or GUI is based on a region or section of Z-web displayed in 3-dimensional or 2-dimensional space or coordinate, in one example. The storage of the Z-web is done in relational databases, in one example, to store node parameters and branch parameters and values, which can be fuzzy or crisp or based on natural language, e.g., “small”, e.g., to describe the length of the branch.
To insert some nodes, in between 2 nodes, one can break the branch connecting the 2 nodes, and insert the piece or node in between, and add 2 new branches to the beginning and end of the added piece, to connect to the rest of the Z-web to expand the Z-web, if needed. The reverse process is applicable, for elimination of a node, if the concept or object is not applicable anymore (e.g., a species of an animal is extinct in year 2020, and the node relating or describing the current live species on planet Earth described in the Z-web has to be updated and eliminated).
Two (or more) Z-webs can be combined, as well. For example, if they do not have any common nodes, the combination is just the simple union of both, with not much adjustment. However, if they have some common nodes (e.g., object “animal” being present in both Z-webs, as a common node), the common nodes can be overlapped together, as a single node, and the branches for a common node can be added from one Z-web into another Z-web. After that, any other node or branch automatically follows the old connections they had from the original 2 Z-webs. However, in one embodiment, we make an adjustment on the values for nodes and branches for the overlapped common nodes to make them compatible. For example, all values can be normalized based on the value of one node on the first Z-web, with respect to the corresponding value of the same node on the second Z-web (mirror node), or ratio of those two values applied to all the values on the second Z-web, to “normalize” the second Z-web, with respect to the first Z-web, to make them compatible.
In one embodiment, we make the adjustment on the node, based on the reliability factor, or other factors mentioned in this disclosure. For example, the value of the first node on the first Z-web is changed towards (or changed to) its mirror node on the second Z-web, if the second Z-web has more reliability factor corresponding to that node. The change can be straight and exact assignment of the value of the mirror node, or can be gradual or partial adjustment towards that value, which could be a fuzzy concept by itself, for example, “almost the value of mirror node” or “90 percent of the value of mirror node”.
In one embodiment, one party makes a first Z-web, and then combines it with N other parties producing N other Z-webs, as described above, to increase the knowledge base and relationship base, including reliability, credibility, truth value, and other factors mentioned elsewhere in this disclosure. This also takes care of the contradictions and inconsistent results, to fix or find anomalies or biases or other parameters described in this disclosure.
As time passes, the size of the super-Z-web increases, and its value grows, as more people or entities contribute to that super-Z-web, which includes more concepts and objects. If all branches associated with a node is broken, the node becomes irrelevant, and can be eliminated from the Z-web. If a node is accessed a lot, its “popularity” value goes up, making it harder to break the branches later. If a value is confirmed or approximately confirmed, in a fuzzy sense, then the reliability of that value increases.
The branches between nodes are not-directional, in one embodiment, so that the relationship is e.g., bi-directional or symmetric. For example, if object A is close to, or located close to, B, in terms of Euclidean distance or meter or length, then B is also close to A. Thus, relationship between A and B is symmetric in that respect. However, in another example, the relationship of “ownership” is not symmetric between a “car” and a “person”, because a person owns a car, but not vice versa. Thus, “ownership” is a directional relationship or branch, between 2 nodes. See e.g.,
In one embodiment, we have hunches or random guesses, or using guided templates, to follow some scenarios, to guess and validate some relationships between the objects. The rules are used for validation of the hunch or guess, e.g., using game theory. For example, one relationship between 2 people in a picture is father-son relationship, as a guess, which can be suggested and followed up to validate that guess, if it is true. If the parameters are non-crisp, then we use Fuzzy logic and sets and boundaries and values. If the assumption or guess ends up with contradiction, we back track, to invalidate the original assumption, and try another guess.
Of course, if later on, we have an input from social network (e.g., FACEBOOK® or LINKEDIN®) or family picture or family album web site or matching family names (or maiden name) or ancestry-type web site, that 2 people may be related, then we do not need to try the guess scheme, and the discovery goes much faster.
In one embodiment, to update a first object, which is based on one or more second object(s), the system tags the second object(s) or node(s) in the Z-web, so that if there is any changes on the second object (e.g., above a threshold, or any changes whatsoever), then as a trigger event, it would trigger the Z-web to ask the calculation module or the processor module to recalculate the first node and all its properties, including all its Z-factors, and optionally let the user know about the update event, for the user to extract data from the first node again, if desired. In one embodiment, it also propagates the update to the other nodes in the Z-web, or other related Z-webs. In one embodiment, this is used for incremental or small changes, or for fast update, or parallel updates in various regions of the Z-web (regionally or in small scale e.g., around one node only). In one embodiment, we have a Z-web with multiple nodes connected to each other, based on the relationships and functions, with different strengths or closeness for each branch connecting the nodes, each having its own Z-factor, including reliability factor and other factors discussed in this disclosure, with each node representing an object, concept, emotion, status, mood, mode, state, subject, number, human, animal, car, table, face, name, birth date, event, or the like.
Accessory Models:
Now, let's look at the accessory models for humans, animals, objects, faces, eyes, or other body parts, for image recognition. For example, for a human, the person may wear or carry a pair of glasses, hair piece, hat, beard (real or fake), moustache (grow or wear, fake or real, e.g., with different styles or sizes or forms or shapes), ski mask, eye patch, sun glasses, scarf, big loose rain coat, turtleneck clothing, body or face covers, umbrella, other accessories, and the like. These alter, modify, cover partially or fully, or hide the features (e.g., curvatures or contours or markers) of the body, face, human, or animal, in a way to make it harder or difficult to make proper or correct recognitions or classifications.
To overcome this problem, we can do multiple solutions. First method is to extrapolate or interpolate the regions, colors, texture, or lines in the image, to fill up the gaps or missing or covered part(s). There are multiple ways to do this. One is to filter or un-sharpen the image or lines to fill the small gaps. Another way is to distinguish the regions or open regions or connected regions, to copy the texture or color from one and paste and extend the patches or small copied regions into the neighboring connected regions, to fill up the gaps with correct color and texture, as much as possible.
Another method is to first add (for example) a pair of glasses to the picture of a specific/same person, by superimposing the picture/image of a pair of glasses on top of the person's face/person's image, and then to compare the resulting image to the images with pair of glasses, for proper recognition or verification for the face or person.
Another way is to compare only the visible parts with corresponding parts of the target images, to find the matches or degree of matches, and treat the invisible parts or hidden regions as “DONOT CARE” regions or “NEUTRAL” regions, which do not contribute to the match analysis.
Another way is to first use a template or generic face with glasses or a model with glasses or an accessory model (e.g., using real image of mannequin, or computer generated model or mesh or 31) surface, or averaging the normalized coordinates of thousands of images of the face), to modify or morph the first image, to compare the modified image to the second image, for match analysis and recognition
Another way is to use neural networks for training purpose, with a large set of faces with eye glasses (for example), so that the neural network is trained (with weights set) to distinguish a person with glasses, or distinguish a specific person with glasses (e.g., with any style or type of glasses, or even when the person not wearing the glasses). In that case, a person with many types of glasses can be photographed for input to the training module, for training the neural network. See e.g.,
In one embodiment, one can model different glasses as a 2-D (2-dimensional) object, and superimpose on the first image/picture. In one embodiment, one can model different glasses as a 3-D object, and superimpose on the first image/picture. The 3-D model is more computing power intensive, but has the advantage of better perspective and more realistic views from different angles. In general, any accessory on human, face, animal, automobile, or other objects can be modeled in 2-D or 3-D model, and stored in one or more databases, for usage, superimposing, editing, replacing, morphing, converting, or adding to the image or model of another object, e.g., human, face, animal, automobile, or other objects.
In one embodiment, one models different glasses separately and models different faces separately, and then superimpose them together. In one embodiment, one models different glasses and different faces together, as one object. In one embodiment, one models the objects using a real faces and real glasses (e.g., pictures or images from real objects), by classifying them, using many training samples, and having at least one of each classification stored for future referral. For example, if we have Nf different types of faces and Ng different types of glasses, then we will have (Nf Ng) different types of combination of faces and glasses. Similarly, for M objects, we will have (N1N2 . . . NM) different types of combination of those M objects, stored in the database.
In one embodiment, one models the objects using a real faces and computer generated glasses types. In one embodiment, one models the objects using a computer generated face types and real glasses. In one embodiment, one models the objects using a computer generated face types and computer generated glasses types.
In one embodiment, the computer generated types are based on real images of real objects, as well, which are classified as different types by the computer, and an average or typical sample is stored as an example of that specific type in the database. In one embodiment, the storage of the example is either analytical, e.g., mathematical formulation of curves and meshes, to mimic the surfaces in 3-D, or brute force storage as a point-by-point storage of coordinates of data points, in 3-D (x, y, z) coordinates.
Features In Images (or Other Subjects) are Recognized In Different Orders:
Let's assume we are showing a picture of a red 2-door convertible Ford car to multiple recognizer units or modules. In the first order or step, they all may recognize the car in the image. Then, in the second order or step, they may recognize red color, or 2-door, or convertible, or Ford brand. Thus, based on the background or experience or training of the recognizer units or modules, the next step may be very different for the recognition process. Eventually, all or most of the features may be obtained by most or all the recognizer modules, but in very different orders. So, there is no universal classification or universal correctness in recognition or absolute classifier or single answer or single recognition method or formula or procedure. Having said that, however, one can still get to the same answer from different routes, e.g., saying or recognizing or resulting in: “a red 2-door convertible Ford car”, or the combination of the following features:
CAR→
-
- +RED+(TWO-DOOR)+CONVERTIBLE+(FORD BRAND)
The principle mentioned above is applicable for any other recognition or any other subject or object, e.g., voice recognition or music recognition.
Recognition Method In an image, for Background and Foreground:
First, for example, we find the background in the image, such as sky or water. That also determines the direction and angle or tilt adjustment for the image. See e.g.,
In one example, once we know the background, all other objects blocking the background, e.g., ocean or sky, will be foreground objects, e.g., boat or airplane, located or positioned in the ocean or sky, respectively. The foreground objects can be recognized from the database of objects, using object recognition module or device, as boat or airplane, and tagged accordingly after recognition process. The recognition can be based on silhouette or shape or shadow or profile or boundaries of an object with respect to the samples of the database, as the percentage of match, between the normalized objects, for faster and better comparisons, using a vector or chain piecewise comparison, or pixel-by-pixel comparison for the silhouette or shape or shadow or profile or boundaries of an object.
In one example, we remove the foreground, and we end up with patches or regions of background objects. For example, once we distinguish a man on the foreground as foreground object, we can remove the man from the image (ending up with a blank region), and end up with the 2 pieces of sofa that the man was sitting on, on the left and right sides of the image. From the texture and color, or continuity of the border lines or extension/direction of the border lines, of the 2 sides, we recognize that they belong, or most likely belong, to the same object. So, we fill up the blank region behind the man's position in the image with the same texture or color from either or both sides of the image (or use an average or mixture of the texture or color from both sides of the image). So, now, we end up with a whole sofa, which is much easier to recognize as one piece, or single region object.
Also, the fact that we know we are in a home environment or context helps us to narrow down to about 200 objects, for example, in our database, related to possible objects in the house, or belong to a specific person living in Canada (based on the conventional furniture for that part of the world, e.g., a sofa, or e.g., knowing a specific person originally from Middle East, with big traditional pillows on the sofa, as possible choices to search or compare for, from the possible-choice-databases, available or compiled by us, beforehand). See e.g.,
In one embodiment, we can recognize the two sides of sofa as a single object, i.e. sofa, without filling up the gap or blank region(s) with color or textual patches using copy/paste routine explained above for small unit regions or patches or tiles, which can be square, rectangle, circular, or non-geometrical shapes, repeated until the whole blank region is scanned or filled up or painted. For example, we extend the boundaries or border lines from both sides to connect or complete the whole sofa border line, or approximately find or complete the border line, to find the final shape of the sofa, to recognize the object as possible sofa. The approximate line can be connected and recognized as one single line, when the line is thickened with a thickness of 2 to 10 points, or more points, to produce a continuous line (thick jagged line). See e.g.,
In one embodiment, we assign a value of “I DO NOT KNOW” to the invisible part of the sofa, in which we try to find the fitting objects based on the visible parts of the sofa, from our library of possible objects, common for a setting, with the condition that on the back (where it is hidden), it can be anything. So, we calculate the reliabilities based on this scenario, and we use fuzzy values to describe this, in one embodiment. In one embodiment, we use Z-web for this purpose, with all corresponding Z-factors.
Adjusting the Tilt or Orientation:
The orientation of an image, such as from horizon line, or water or ocean line far away, or tower in background, which indicate horizontal line or vertical line in the perspective view or expectation of humans, indicate how much an image should be rotated or tilted to get into the right orientation. For example, that normalizes the head or face of a human to be in the right angle or direction or orientation, to pre-process, before the actual recognition of the face or head by the face recognition module. That increases the accuracy of the recognition at the end, for the objects at the foreground.
To Find a Continuous Line:
To find a continuous line, one searches for the next point in the line, e.g., black pixel or dot or similar color pixel, in left, right, diagonal left up, up, down, diagonal left down, diagonal right up, and diagonal right down, i.e., in all eight neighboring directions or pixels, to find any match, which produces continuity in the line, point-by-point, to extend the line.
For small discontinuity e.g., due to bad image quality or a copied image by old copy machine multiple times, the bridge gap of 1-2 pixels can be forgiven or filled up. Thus, the search is beyond the immediate neighboring pixels, going e.g., to the 3rd neighboring pixels, to find a match of pixel, to assume continuity for the line, and filling up the gaps with the same pixel or dot, to get a final continuous line. Or, one can defocus or widen the lines, using a filter to reduce the contrast for the edges, to bridge the gap of 1-2 pixels with filled pixels or dots, to get a final continuous line.
To find a narrow boundary or border, from the thick jagged line mentioned above, one can get the average coordinates or select the middle points of the thick jagged line, as the final fine boundary, which defines the object, e.g., sofa, very well, with sharp boundaries, for easier detection or recognition, versus the object with thick boundaries, which is harder to detect, when the small features are hidden or overshadowed by thickness of the line, itself.
Another way is to use skeleton or bare bone shape, to simplify the shapes fast and efficiently, as the first-cut/coarse search and comparison, from database of skeletons or shapes, to name the objects or tag them, which accompanies the objects as comments or tags data from now on, after tagging.
Another way to find or recognize a sofa is to use DONOT CARE or NEUTRAL region assignments for the blank region, for partial matching with test images as targets, to find the similarities between the object and targets based on the visible parts, and treating the invisible parts as having any values possible (or ignore them, as having no or negligible weights), for comparison or recognition purposes. The final match score or probability is only or mostly based on the visible parts, with respect to target or possible-object databases.
Use Images from Different Angles or Perspectives:
To model an object, from a 3-D perspective, one models the object using images taken by a real camera, from different angles. For example, for the recognition of a face or person, one looks at the face from multiple directions, e.g., from side view left, front view, half-side view fight, and back side. Thus, we store the multiple views from different camera positions or angles, for the same person, for later recognition of the person, to find an exact match or a match between two or more of these snap shots or images (i.e. using limited numbers of images, as discrete sampling, for continuous matching positions, later on), as interpolation or extrapolation of one or more images, or some weighted average of them, or some average of them.
Use Computer Models of Objects:
Also, one can use a computer generated model for N possible shape of heads for men, women, and children at different ages, for various ethnicities and races, based on the images of shapes of head taken and input them to the model (e.g., artificially rendered or calculated or constructed by a computer), to cluster and classify all possible head shapes on the planet (where N is usually a manageable number, say, e.g., 100). So, starting from a 2-D image of a new person's face or side-view (person P), it can trigger or match approximately the image of one of those N shapes from the head shape library, and thus, call up the corresponding model for the selected head shape from the library or database.
Now, in one embodiment, we have a correction that can make the model more accurate. We change the parameters of the head on the model slightly, to match the image of the face or head for person P exactly, from the 2-D image, using manual adjustments, or computer control or automatic adjustment, e.g., fuzzy rule based adjustment, to morph one to another, so that the contours and/or points on the mesh representing the face or nose or head match exactly with the model. The morphing mechanism details are described elsewhere in this disclosure.
In one embodiment, we have a correction that uses two or more of those N shapes (say, e.g number of those N shapes) from the head shape library, and then combine them to get an approximate match, e.g., using a linear combination of them, or weighted average of them, or take an average of them. Then, in one embodiment, we have a further correction, similar to above, to change the parameters of the head on the model slightly, to match the image of the face or head for person P exactly, from the 2-D image, using manual adjustments, or computer control or automatic adjustment, e.g., fuzzy rule based adjustment, to morph one to another, so that the contours and/or points on the mesh representing the face or nose or head match exactly with the model. The morphing mechanism details are described elsewhere in this disclosure.
In one embodiment, we have some or all of the N shapes sub-divided into Q1 to QN shapes, respectively, as subcategories, for minor differences between head shapes. Then, we have better matches based on subcategories. However, the overhead for storage and computation is much higher, since we are dealing with much higher number of shapes now. That is, we have now: (Q1+Q2+ . . . +QN) shapes, rather than N shapes.
In one embodiment, we adjust the mesh or points or contours representing the face or head, as an example, using the parameters that change the coordinate of points, or change the formulas for curves or family of contours, in the computer model, and changing those parameters by a small percentage or small relative deviation. Then, we observe the result: If the difference (e.g., sum of square of differences, or sum of absolute value of the differences, as error function) on the match for all points or contours or meshes with respect to the model for the s sleeted shape gets larger (gets worse), then we should change in the other direction or change other parameters. If the difference on the match for all points or contours or meshes with respect to the model for the selected shape gets smaller (gets better), then we are on the right track, and we can continue on the same direction, until we get worse off. Then, we stop at that point for that parameter. Then, we try other parameters, one by one, or in a batch, or bunch together, to optimize for complete match with the model. That is, we use a feedback to adjust the parameters, for complete match, as much as possible.
In one embodiment, to adjust the difference value mentioned above, we may be in a local minima region of the optimization curve for the difference value(s) function mentioned above, and small adjustments may get us only into a local minima. However, to get into an absolute minima of the optimization curve for the difference value(s) function mentioned above, one has to get out of the local minima region. To do so, we need a random adjustment on the parameter(s) or big adjustment on the parameter(s), to land in another part of the optimization curve for the difference value(s) function mentioned above. That will increase the chances of getting out of the trap of being in a local minima region for all optimization adjustments at all times.
Of course, even if we get to local minima, rather than absolute minima, for optimization, we still may have a good result for match process, to stop further search and optimization or adjustments, as mentioned above. That can be checked using a relative or absolute value as threshold, or an incremental improvement analysis, to stop beyond a threshold, for the optimization process, as optimization any further would not worth the cost of more computation power spent on such incremental improvements, if any.
Look for Expected Objects:
For example, in an office environment, one has a list associated with a typical office or law firm office or dental office, which are stored as possible objects in the office, in a web of related objects, or databases, related to an OFFICE or DENTAL, OFFICE. So, an object behind a person in an office on the table may be a fax machine, which is a possible target candidate for examination and image recognition comparison, obtained from the corresponding list of related objects for the OFFICE. That increases the reliability, speed, focus, and accuracy of the recognition process.
One can also re-use the related objects from one into another one. For example, an “office” is superset of a “dental office”, for most cases. Thus, all properties of “office” are a subset of (and included in) those of a “dental office”, including e.g., related objects or expected objects or owned objects or property objects. That is, they inherit each other's properties automatically. See e.g.,
Of course, in one embodiment, these concepts above are all fuzzy concepts and sets, with no hard or crisp boundaries, and with qualifications e.g., “for most cases” or “usually”. Please see the discussions elsewhere in this disclosure, regarding the handling and processing of these concepts, values, and parameters.
OCR, as Textual Information, to Help Build the Relationship Web Between Objects:
In the next step, as one example, we look for a text as an object in the image, to recognize, for example, the brand, model number, and the type of the object, e.g., HP printer Model Number 100, written on the object, as text. So, we invoke an OCR (optical character recognition) module to read the text, to find and search for more relationships between the objects in the image. The text in the image can be vertical, slanted, wavy, morphed, or curved, as in a book in a bookshelf, or as in a newspaper on a table at an angle to the camera or frame of the picture or image, or as in a word written as a motto or slogan on a flying flag with the wind power behind it, or as a reflection of some big poster on the side the building or highway reflecting the text image on a wavy water or pool or pond nearby, or as a security word for user authentication (against sniffing search bots) with a slanted or twisted image of a text, usually with no meaning, on the screen or monitor of a computer.
List of manufacturer and model numbers or the like are also listed in separate files or databases for search and matching or recognition or validation, to further limit or focus or specify the identification of the object, such as printer or fax machine in the example above, using the OCR as a tool or as one of the linking methods between the objects.
On the related objects, e.g., once a computer is determined as an object in the image, we can expect a possible mouse or monitor (with some degrees of certainty corresponding to each device), or with some membership function or value associated with a fuzzy membership for mouse as an accessory to a computer, and hence, expecting a mouse as an expected object nearby in the image, and thus, look for it as a target object around a given computer, from a database or list of possible objects in the neighborhood.
The Distance or Size as a Factor:
In one embodiment, the distance to the object s also taken into account, for example, G-meter or feet, for estimation, for proximity or location analysis, as the search radius and location estimator, e.g., based on the center of the object, e.g., based on the estimated scale of the object or picture, or relative to the size of the neighboring objects, or typical size of the objects. For example, the mouse's length is about 20 percent, or 20 plus/minus 5 percent, or exactly 20 percent, of a length of a laptop, or a specific laptop, or typical laptop, or an average laptop, or for a range of laptops, obtained from our database for relationships between the related objects, e.g., laptop and expected nearby possible mouse, with its corresponding membership value and reliability value and expectation value, relating the 2 objects, from A to B, as 2 nodes, in the network or web or relationships, e.g., for distances or sizes. Another example is the typical distance between a laptop and a mouse is 1-5 feet, for possible search location possibilities, for the center or the edge of the object, e.g., mouse. See e.g.,
For some examples, for fuzzy ranges or fuzzy values for distances, we use unfocused or fuzzy lines or fuzzy geometry lines, with fuzzy distances and fuzzy endings and fuzzy thickness, for geometrical representation in Z-web. For some examples, for crisp ranges of distances (or uncertain distances with error values), we use dotted lines around the average value or around the minimum value, for geometrical representation in Z-web. For some examples, for geometrical representation in Z-web, we can draw spheres or circles, for radius of search of a target object, with respect to two or more given objects, and from their intersections of the regions or overlaps of areas, we can further pinpoint the location or distance of the target object.
Note that the size of the object is estimated or determined by comparing to relative size or distances of other objects in the image or video frame, as typical values in the library, or as a value we already know for a specific object, e.g., Fred Jackson's height is 6 feet. It can be based on Fuzzy parameters and values, as well, e.g., Fred Jackson is very tall. The perspective or depth in the image can he estimated using rectangular objects, such as table, having merging boundary lines for parallel sides, by extending the border lines to the back of the image, so that they cross at an imaginary perspective point IPP in the background of the image, which indicates the perspective for the image with a point located at infinity, very far away. Note that IPP can generally be more than one point or a line, representing infinity, or far away, in the image, at the horizon. Then, from that, the relative size or distances or angles can be obtained, using simple geometry relationships, e.g., mapping the distances or other lines as a projection on the imaginary lines connection to IPP (called LPP), or as a projection on lines perpendicular to those LPP lines (called TPP), which are generally curved or circular shaped lines or family of lines with the center at IPP, in the perspective coordinate system of the image. For example, we divide the image into family of LPP and TPP lines (or curved lines), with some distance between each of 2 neighboring family members, to cover the image like a tilted “floor tile” scheme, and then for each dimension in the image, we try to do the comparison with the dimensions in the same neighborhood with known sizes, e.g., from known objects, e.g., we know that Mark is 6 ft tall, and that gives a reference size for objects in his neighborhood in the image.
See e.g.,
b=V cos(E)
a=V cos(G)
Now, we want the ratios, to some known values or objects, e.g., as shown on the highlighted rectangle in
acalculated=(a/a1) areal
bcalculated=(b/b1) breal
LV=(acalculated2+bcalculated2)
In another embodiment, note that for TPP, we have to find the distances on the curved lines, e.g., a piece of a circle with a radius on LPP, originating from IPP, and ending at the point of interest (at the intersection of those specific LPP and TPP).
In another embodiment, the projection of a line SPP with a length GPP on the LPP line is mathematically given as, PPP:
PPP=GPP·cos (APP)
Wherein APP is the angle between that specific line SPP and a neighboring LPP line, to project on the LPP line. Once we have the projected lengths on those specific LPP and TPP, we can compare that with other projected lengths from known objects with known sizes in that neighborhood, as projected on the same nearest specific LPP and TPP, to get a relative distance or size, or ratio, to get the size of the unknown object (approximately).
In another embodiment, instead of using projection values, as shown above, one simply compares the size of the line piece from the unknown object with the size of the line piece from a known object, in the same neighborhood, to get the ratio, and then, get the size of the unknown object (estimated). Of course, the smaller the meshes associated with LPP and TPP, on the image, the more accurate this estimate will be.
Note that in the general case, going from A to B may be not the same as, or reversible, with respect to going from B to A, e.g., between mouse and laptop as 2 related objects in the relationship web, with respect to the values of membership value and reliability value and expectation value. Thus, we can show that by two arrows going from A to B, and from B to A, with different strength or thickness or width or length or size, signifying the various valuations of membership value and reliability value and expectation value, in different directions. For example, in some embodiments, the expected value of finding a mouse in a specific region or radius or coordinate in the image (given a laptop is found nearby, as an assumption) is different from its reverse situation, i.e., it is different from the expected value of finding a laptop (given a mouse is found nearby, as an assumption). See e.g.,
In
In other embodiments, the two directions are reversible and have the same values for both directions, e.g., for membership value and reliability value and expectation value, between 2 objects, e.g., mouse and laptop.
Now, having specification or range of expectations, for possibilities and probabilities, for example, for distances and sizes, one can search more accurately for the secondary object, e.g. mouse, around a given laptop in the image, or in the universe around us, as the primary object. For example, given a distance between centers of 2 objects, as 0-5 feet, we can design a circle around the primary object, with that radius of 5 feet, to define a region for possible existence of the secondary object, e.g., mouse. That would reduce or limit the search time and criteria, or increase accuracy for a given time and computing power.
The radius can be defined in 2-D or in 3-D space in the image, depending on the fact that the second object has or may have any support for standing in space outside the planes defined by the first object, e.g., having a tripod or legs or support for a camera or mouse. In the 3-D space, it becomes a sphere, with radius R (instead of a circle or projected circle), which has a cross sectional projection or view on the image as an ellipse or oval or curved region, depending on the point of view or perspective view of the camera or image or user. The region defined by circle or sphere, or their projections on the 2-D original image under study, signifies the possible locations allowed for the center for the second object, e.g., for its coordinate(s) or center of mass or corner(s).
Position is also a factor for building relationships between objects, as for example, the 4 legs of a table, with respect to the table, which are usually presumed to be located and also attached at the bottom of the table (unless the table is reversed or broken, e.g., in a fight scene or war scene, as an example, depending on the context or history or assumptions, beforehand, which can change some relationships drastically, as in the case of the image of a war scene or hurricane scene disaster). The position or relative locations are defined using directions or distances, e.g., up, down, diagonal up, 45 degree up left, 5 ft, top, bottom, side, corner, and the like. Note that most of these concepts are fuzzy concepts, useful for membership values, e.g., side of a laptop, or corner of a laptop.
As mentioned above, the context is also very important. Given an image of a war scene or hurricane scene disaster, one may expect to find a table reversed or with broken legs, opposite or contrary to any normal expectation or relationship between normal objects in a normal environment. Thus, the relationship web is very different for those situations, with respect to normal situation. In addition, that is one way to confirm that an image is possibly from a war zone, based on tables with broken legs or houses with no roofs on the top. See e.g.,
In one embodiment, when we look at a picture, we focus in the middle or at the main feature(s), first, as e.g., indicated by histogram or contrast map. Then, we look for other expected objects nearby, using the related objects list with associated probability and associated expected distance (relative or absolute values), which is part of Z-web. In one embodiment, once we find e.g. a face in the image, we can assume that most likely that other faces or other eyes or similar objects, if any, in that image, are in the same scale or distance or order of magnitude, which can adjust the scale or size of the basis functions, such as wavelets, to find the other eyes or faces in the image much faster, focusing or using only basis functions or filters within similar or same scale for basis functions or object size. In one embodiment, when scaling the basis functions, the lines or curves defining the basis function has the same thickness as that of the original. In one embodiment, when scaling the basis functions, the lines or curves defining the basis function get scaled linearly with respect to that of the original. In one embodiment, when scaling the basis functions, the lines or curves defining the basis function get scaled non-linearly with respect to that of the original, e.g. based on exp(x), log(x), or x2.
Going Backward (and Testing or Verifying) on Assumptions:
As we get the input and build our web of relationships between objects or concepts or subjects, e.g., emotions, humans, and tables, we add reliability, truth, credibility, and consistency of the information, which can be addressed by Z-numbers or by fuzzy logic membership or other fuzzy concepts or other reliability calculations, also described in the U.S. Pat. No. 8,311,973, by Zadeh, which addresses Z-numbers and its applications, as well as other fuzzy concepts, plus the “trustworthiness of speaker”, “sureness of speaker”, and “statement helpfulness”, with the analysis for cascaded or network of information sources ending up with a “listener”, e.g., in
Now, in one embodiment, let's start with multiple assumptions, A1 to AN, and from there, we can get some web connections for relationships between M objects, subjects, words, and concepts, e.g., emotions, humans, policeman, teacher, dog, and car, in this relationship web, as nodes on the network. All the relationships and assumptions have reliability, truth factor, confidence level, and credibility metrics (with their corresponding membership functions or values).
Now, in one embodiment, we start from a node and continue building the network, until we get to a point that inconsistency or contradiction flag is up, in terms of property of a node which gets contradictory results from different sides or routes. Then, we backtrack and clean up the route to the original assumption(s) or node(s) that may have caused this problem, to remove or change the assumption(s). We can change the assumptions one at a time, and see the results again, until “satisfied”, which is also a fuzzy concept (for the degree of “satisfaction”). Or, for N being a very large number, we can change multiple assumptions at the same time, and observe the results, to adjust the assumptions in a feedback loop manner, or based on some fuzzy rules.
In one embodiment, for conditional relationships, or multiple choices, we can continue, until we get to a dead end or conflict, and then, backtrack to eliminate or adjust one or more choices, on the chain going backward, to correct or adjust some assumptions, choices, or conditions, on the way.
In one embodiment, using assumptions on human emotions, one can do behavioral analysis on individuals, or collectively on whole society, e.g., how the people feel or react on a bad news, such as earth quake, using e.g., the sad faces in images, or text analysis on expressed or typed words such as “Disaster!” in the email or texting message on phone. The collection of nodes in a Z-web can indicate that a person is very angry or sad at a given moment.
Of course, as the mood of a human changes, the corresponding Z-web changes accordingly, with new nodes, weights, reliability factors, and the like. So, the Z-web is a dynamic structure which is potentially time-dependent, with a corresponding characteristic time period (TChar). For example, a geographical factual Z-web with many constant facts at its nodes has a large TChar, because we do not need to update or change that Z-web very often, as most of their values stay the same for a long time (versus some Z-web related to e.g., the stock market, with fluctuations and variations on a daily or hourly basis, which requires daily updates, and thus, has a lower value).
Optimization of Quality of Different Aspects of Image:
Consider the line on any line drawing image. The thicker the line, or the wider the tip of the pen used to draw the line, the less features are visible from the line drawings, as the small features are dominated or lost by the thickness of the line, itself. Sometimes, for some applications or situations, we want to increase the width of the lines or boundaries, for the sake of having continuous boundaries or borders between different objects, for better object recognitions or discriminating between neighboring objects, to figure out what is in the picture or image. However, for any image with small tiny features, that increase in the width of the lines or boundaries may cause problems of wiping out or hiding or losing the small features for the borders of objects or regions in the image, if those features are important for any other analysis. So, we have to figure out at the beginning that which one is more important, to preserve one or the other, i.e., in favor of one or the other. Or, we have to figure out at the beginning that to what degree this process should be done, in favor of one aspect, before damaging the other side/aspect.
So, (i) we classify the images at the beginning, and (ii) also see what kind of interest or information or query we need or want from the image(s). These 2 parameters determine how far we should optimize the image, for which aspect, and in the expense of what other aspect of the image. The compromise factor between different aspects of the image and optimization factor for each aspect of the image are also fuzzy parameters, and can be determined using a fuzzy rules engine or a fuzzy optimizer. The fuzzy rules engine or a fuzzy optimizer are explained here in this disclosure, as also explained in U.S. Pat. No. 8,311,973, by Zadeh.
One way to avoid this analysis or compromise is to make 2 copies of the same original image, and then optimize the first aspect on the first copy, and optimize the 2nd aspect on the second copy, and then extract information from each image or copy separately for the 1st aspect and the 2nd aspect, from the image or copy and the 2nd image or copy, respectively.
Another way is to make one analysis on the first aspect from the original image (that does not need much optimization or correction on the image), and then change the image to optimize the 2nd aspect, for analysis of the 2nd aspect, to extract more information about the second aspect. This way, we get somewhat good information about the 1st aspect of the image, and excellent/large amount of information about the 2nd aspect of the image. Yet, the overhead about computation power or storage of images is not as large as the previous solution, given above. So, it is a kind of middle ground compromise solution, good for some applications, which need some accuracy, but at lower cost for computation and analysis (or shorter turn-around time for analysis and results).
Window for Examination:
When looking at one image, for one embodiment, if the window for examination of the image is too wide, and we get one signal from all of the window, then we may get the average values from all regions of image contributing to the result. Then, in those situations, we may not get some of the features from the image. For example, if the features are based on sinusoidal function (sin(x)), with half of the time negative and half positive, in 2-D space of the image, then the average for all regions, containing a lot of the periods for the function (assuming small periodicity for such sin(x) function, i.e. small T, for this example), would be zero or near zero, for the total average. Thus, the behavior of sin(x) for the feature in the image is not detected at all, in this example.
Now, if the window of examination is too narrow, and the changes are negligible for consecutive windows, in absolute values or relative values, then the process is too slow or expensive for analysis, and we may also miss detecting some of the big scale behaviors in the image. Thus, the optimum window size depends on the periodicity (TF) and size (LF) of the features in the image, to have both efficiency and accuracy for the image analysis. So, at the beginning, we classify the image based on those parameters (TF and LF), plus its complexity (CF) and concentration of features (MF) that we are looking for in the image. Then, the size of the window (SW) is determined from all those parameters. Note that all these parameters can be expressed by e.g., real numbers (fuzzy or crisp values) or in terms of human natural language, e.g., “large window” (fuzzy values)
For example, we have TF as 2 features per 50 pixels or 2 features per cm2 or 2 features per 5×5 pixel square or 2 features per cm of boundary. For example, we have LF as 50 pixel or 5 cm or 5,2 times bigger than size of the mouse of the computer or “bigger than size of mouse of the computer” (as fuzzy value).
For example, in one embodiment, we have complexity (CF) defined as the number of gray scale used (out of 256, for example) (or available) in this particular image, or defined as number of color values used for components of RGB or CMYK system in the image, or defined as the number of intensity values used (out of Q total values available) for the image, or defined as the percentage of variations, in diagonal or horizontal axis, in the middle of image or passing the center of the image, in the intensity of pixels, plus the directions of those variations in the pixel intensity (which can be large or small positive or negative numbers or percentages or relative values), or expressing any of the above per square pixels or square cm or unit of area, or similar definition as a metrics for the complexity of an image.
For example, we have concentration of features (MF) as number of features (or spikes or crosses or knots or curves or small squares (as examples)) per square pixels or square cm or unit of area, as examples, or when the features are confined on a line or curve or boundary, MF may also be expressed per pixel or cm or unit of length. For example, we have the size of the window (SW) as 100 by 100 pixels, or 2 cm2, or twice as big as the mouse of the computer in the image, or “very very large” (as fuzzy value), or 1 percent of the whole image, or “small square”.
For example, in one application or embodiment, for small TF and small LF, plus high complexity (CF) and high concentration of features (MF), the size of the window (SW) is set to be small, e.g., 3×3 pixel (square).
In general, we have a function FW, defining SW as dependent on parameters:
SW=WW(TF, LF, CF, MF)
Extracting Clues and Information from Images, to Determine Relationships:
From an image, picture, video, drawing, cartoon, caricature, sketch, or painting, one can guess or estimate or find relationships or find attributes or find the degrees for relationships or find connections between objects, subjects, humans, animals, plants, furniture, emotions (which can be used to predict e.g., social behavior, purchasing behavior, voting behavior, or rating system behavior), ownership, properties, characteristics, or the like, related to, for example, the following:
The age of the subject or person or animal, ethnicity of a person, relationships between subjects (in a picture or painting or image or video frame), picture setting (e.g., at office, official, military', family gathering, class reunion, primary school student picture, graduation from college, prom dance event, black tie event, Olympics medal ceremony, Oscar Academy Awards event/night, or karate class), family membership, happiness (or misery, despair, anger, emotion, or mood), closeness (friendship, or how close the subjects are to each other), intelligence of the person, sophistication of the person, gender of the person, style of the person, location of the picture, year (in which the picture was taken), political affiliation, country (in which the picture was taken), language of the location (in which the picture was taken), time of the day (in which the picture was taken), season or month, special occasion (New Year celebration at Times Square in NY City, Christmas, wedding, or carnival), special location (Disney Land, cruise trip, on the Moon, Grand Canyon, or near Eiffel Tower), temperature of air (in which the picture was taken), humidity (in which the picture was taken), time zone (in which the picture was taken), altitude or location on the planet Earth (in which the picture was taken), height (in which the picture was taken), depth (in which the picture was taken), or environment (e.g., cloudy, rainy, war zone, or foggy), as some examples, or the like. See e.g.,
The correlation between objects, subjects, and concepts, at nodes in the relationship web or network, as the web grows and gets built up, with more relationships and larger number of nodes, brings more and more objects, subjects, and concepts together, and validates or verifies estimates, guess work, and possibilities, with more accuracy and higher confidence level.
The input to the web of relationships comes from many sources, e.g.: textual information, video, music, noise, voice, still images, pictures, sound bites, expressions, moods, emotions, tags, comments, recommendations, LIKEs on a web site, customer feedback, TWITTER®, FACEBOOK® entries, emails, blogs, votes, political opinions, surveys, summary of data, medical images, weather forecasts, historical data, geographical data, mathematical, physics, and chemical facts, historical monuments, famous quotations, books, slangs, Wikipedia, encyclopedia, dictionary, thesaurus, translation books, county land records, birth certificates, lectures, novels, science fiction, documentaries, history books, magazines, picture albums, databases, private network or storages, class notes, exam answers, dating sites, ancestry web sites, social media sites, petition documents, tax returns (if available), resumes, biographies, biometrics, gene or DNA sequence, medical data, medical history, medical knowledge, chemical formulas, mathematical relationships, physical constants, physical phenomenon, abstract concepts, architecture, psychology, philosophy, proof methodology, inductive reasoning, logic, calculus, hand written notes, scripts, computer program, codes, encrypted message, sign language, alphabet, Internet, search engine, opinion of famous people, opinion of friends, friend suggestions, social media votes or suggestions or opinions, court documents, dockets, or the like.
For example, to find the age of a person in a picture, the number of or concentration of wrinkles on the face or neck or skin can be counted or detected (as the older people tend to have more wrinkles, as an example), or based on the ratio of the size of the head to the rest of the body or height (as the average ratio or ratio changes for different ages, for most people, tabulated based on millions of samples in the database), or features of the face and their dimension ratios (as is different at different ages, within some range, for normal people, stored in the databases, which can be a crisp value or fuzzy parameter), or having bi-focal eye glasses (usually for older people), or having a hearing aid (usually for much older people), or having a bald head or receding hair line (usually for adult people, and usually male subjects), or having only one earring, nose ring, or tattoo (usually for younger people), or having 2 earrings (usually for female, above 10 year old, as an example), or having a tie or bow tie (usually adults in formal settings, and usually male subjects), or having a top hat (usually adults in formal settings, and usually male subjects), or having a baseball hat (usually kids or young adults, and mostly male subjects), or having a beard or moustache (usually 12 years or above, as an example, and almost all male subjects),
Please note that if we have other information about the culture or town or country or the date of the picture, we may be able to determine the age more accurately, for example, in 1960s, a lot of college students in certain countries wear tie in college, but it is not true for college students in US in year 2000. Another example is for Scottish culture or region, we know that Scottish men wear the skirt as tradition, which may tilt the possibility and probability and confidence and reliability of the decision or recognition or classification, one way or another, based on the subject having skirt on, in the picture or image. Thus, the culture and date of the picture and context and traditions and environment may all be influential and factors in the decision making.
Some of the features or criteria or tests mentioned above also apply to gender, as described above, e.g., having a moustache or beard on a person in the image or picture. However, if we know, e.g., that the date of the picture was Halloween, located in US, then the moustache may he fake or on a Halloween mask, or if the location is Universal Studio for a movie, from scene of the movie, then the moustache may be fake. So, again, the context and date of the picture are important to tilt the values or relationship strengths or possibilities or probabilities.
Also, most of the rules stated above are fuzzy rules, for relationships, for various objects and subjects or concepts, such as: “Having hearing aid in the ear, in the picture, usually infers that the subject (the person under evaluation, in the image) is probably an old person”. First of all, “old” is a fuzzy value, and then “usually” plus “probably” can be handled by the Z-number mathematics and algorithms, as explained in this disclosure. In addition, fuzzy rules engine and related algorithms, e.g., backward chaining inference engine and the forward chaining inference engine (for handling a set of fuzzy rules for the relationships that we stated above, for determining the age of the person, as an example), are also explained in this disclosure.
Now, we have more information extracted from the images. For example, the picture setting may be at an office, with indicators such as tie and jackets or formal dresses, as well as desk, round table, water cooler, copy machine, cubicles, partitions, white board, calendar, deadlines on the board, tasks on the board (read by the OCR and understood by the natural language processor, as being tasks with dates in front of them, and possibly some arrows, with time line on horizontal axis), conference room, conference phone, employee award on the wall or on the desk, “men's room” indicated by word or by a “man” symbol on the door to the bath room, rack of coats or hangers, name tags on the desk or wall, room numbers on the door or wall, or the like.
One indicator may not have a high correlation coefficient to a setting, but a combination of multiple indicators has a much stronger correlation coefficient, e.g., some of the indicators mentioned above, with respect to the “office” setting. Also, one “fax machine” may be have a different correlation coefficient or relationship factor with respect to general office, or document processing office, versus dental or doctor office. So, same object in different environments or locations or contexts have different relationship factor, e.g., in day time setting versus night time setting.
To examine a setting, for example:
-
- for official setting, we look for jackets and ties,
- for military setting, look for guns and uniforms,
- for family gathering, look for kids and grand parents or people at different ages,
- for class reunion, look for people of the same age and banners stating a university or high school name, plus graduating date, e.g., 1977,
- for primary school student picture, look for a lot of kids of about less than 12 years old,
- for graduation from college, look for graduation gown and cap,
- for prom dance event, look for prom dress and limousine,
- for black tie event, look for black tie dress and jacket,
- for Olympics medal ceremony, look for Olympics sign and medals around the neck,
- for Oscar Academy Awards event/night, look for Oscar symbol or statue,
- for Karate class, look for Karate belt and outfit, and the like.
These relationships come from expert humans, or many human voting or inputting, or from trained computer learning machine, or extracted from millions of relationships from a huge observation sampling or history file or database. See e.g.,
Other examples are:
-
- for family membership (look for hugging, kissing, how close people stand in a picture, smiling in a picture, casual dressing, vacation setting in the background, similar faces, similar clothing, at a dinner table at home setting),
- for happiness (or misery, despair, anger, emotion, or mood) (look for the shape or expression or size or angle of mouth, face, eye brow, eye, color of face, or lines on the face, based on stick diagram defining laughing or other expressions or moods, or based on formulas defining those expressions, or based on curves defining those expressions, either analytically stored as curves or graphically stored as pixels, or based on thousands of faces stored from real people, tagged for expressions on their faces, for learning samples, as supervised learning),
- for closeness (or friendship) (look for how close the subjects are to each other in the picture, as how many people are in between, or how tight they stand close to each other, or how hands hold each other and where they are located with respect to the body, which can be also trained with thousands of already tagged pictures by people or experts, as one way of doing it, or can be understood based on the relationships of objects, e.g., first person's hand (H1) behind (or covered by, as function CB) the second person's shoulder (S2), indicating hugging, indicating closeness or friendship, or mathematically expressed as, when finding the objects and ordering of objects in the image: CB (H1)≡S2) (Note that all objects in an image can be analyzed, to find which is front and which covers what, with mathematical relationships, as indicated above. Once part of the object, as expected, per our expectation (e.g., from shapes stored in a database for that name or object), is missing in the image, that is an indication (of most likely) that part of that object is behind another object, in that image.),
- for intelligence of the person or sophistication of the person (look for book in hand, standing near a library or concert hall or museum, degree near his title or name in text, titles on the door or email, his friends, his/her family members, her job, her SAT score, her GPA, her resume, her publications, or degrees in the frames on the wall or on desk),
- for gender of the person (look for dress or clothing, hair cut, shoe, accessories, name, size or weight or height value or ratio, habits, title behind the name (such as “Mr.”), favorite video game or movie or actress, and color of choices) (Note that these are still not deterministic at all, same as other parameters and indicators mentioned above. That is, sometimes, stereotypes and generalization are very misleading. However, using combination of all data and relationships and aggregating them using our analysis on our “Z-web” increase the accuracy and reliability of the recognition.),
- for style of the person (look for clothing or hair cut or shoe or glasses or wine choices or drink choices or car or watch),
- for location of the picture (look for monuments or famous buildings or names or landmarks or emails indicating the locations of next vacation or tickets for airline or passport stamps),
- for year (in which the picture was taken) (look for clothing style, text, objects in the background, such as cars or building, hair style, name of famous actors, name of movies on display, the president of the country, or tags or dates on the picture or image),
- for political affiliation (look for tag on the car or bumper sticker or pictures on the wall or affiliations or clubs or friends or geographical area or job or title or essay in school or courses taken in college or food choices or vacation choices),
- for country (in which the picture was taken) (look for landmarks, names, tags, signs, street names, architecture, pictures on wall, language on signs, people's faces, stores, cars, license tags, having snow on ground, type of trees, type of foods, politician names, national hero, famous athlete, famous singer or artist, or TV programs),
- for language of the location (in which the picture was taken) (look for names, tags, signs, street names, architecture, language on signs, people's faces, stores, license tags, or TV programs),
- for time of the day (in which the picture was taken), season or month, or special occasion (New Year celebration at Times Square in NY City, Christmas, wedding, or carnival) (look for Christmas tree, decorations, snow on ground, trees with no leaves or colorful leaves, big clock on the tower, position of Sun in sky, light or darkness, frozen lake, ice fishing, or winter sports),
- for special location (Disney Land, cruise trip, on the Moon, Grand Canyon, or near Eiffel Tower) (look for landmarks, text, or structures),
- for temperature of air (in which the picture was taken) (look for steam or fog, rain, snow, ice, people with bathing suit, beach, ice skating, snow man, sweat on face, Sun reflecting on a shiny metal object, thermometer, thermocouple, or weather channel on TV),
- for humidity (in which the picture was taken) (look for steam r fog, rain, snow, ice, sweat on face, mold, green and dense vegetation, or rusty cars in the street),
- for time zone (in which the picture was taken) (look for location landmarks, country', city, names, text, clock, region, marker on the map, flag of the country, or email time record),
- for altitude or location on the planet Earth (in which the picture was taken), height (in which the picture was taken), or depth (in which the picture was taken). (look for landmarks, signature characteristic, perspective view, or indicators of coordinates or locations, such as cloud in sky or fish in deep sea),
- or for environment (e.g., cloudy, rainy, war zone, or foggy), as some examples, or the like. (look for indicators or signatures, such as fog, cloud, wet street, tanks, soldiers, and ruins in a war zone).
In one embodiment, the information on the camera phones (on its memory, processor, or controller module), or on image (as text), or tagged as a header or file or comment to the image, e.g. GPS (location), date, lens data, focus information, and the like, are used for location of the picture, e.g., specific city and monument, or date, e.g., Jul. 4, 1999, 4:30 pm, or focal length, or estimate of distances in the image, or the like. These can be used for correlation with other objects and within themselves. Thus, we can input this information into our Z-web, as new nodes and Z-factors, e.g. for recognition purposes or reliability analysis.
Different Components of Recognizer:
The recognizer module has many sub-components, to do analysis on text, e.g., OCR, image (e.g., image recognizer), video (e.g., video analyzer), voice (e.g., voice analyzer), music, taste, numbers, patterns, texture, faces, names, records, tables, lists, “big data”, and the like, as input modules, to gather, analyze, and aggregate, to find the relationships between objects and concepts, based on the reliability, confidence, truthfulness, probability, and possibility, as discussed elsewhere in this disclosure, to build the “web of relationships”, which we call “Z-web”, and to find or recognize or validate or confirm other or same objects or concepts or relationships. See e.g.,
For constructing the Z-web, we can use various types of input, to build up relationships as described above, e.g., but not limited to: image, video, text, sound, voice, music, biometrics, table or list, tag, comment, metadata, multimedia or movie, link to information or web site, header, summary or abstract, record or database, listing, matrix, geometrical shapes, symmetrical shapes, patterns, symbols, abbreviations, encyclopedia or dictionary, personal data or preference, historical or geographical data, physical or chemical data, and/or mathematical facts, or the like.
Adjusting Colors:
All colors look the same in a dark room, or in a picture with low light intensity, e.g., picture taken at night with no flash on the camera. So, in terms of recognition of a face, a normalization or adjustment is needed to convert the color or intensity of the pixels for a face in a dark image, to correct the color and intensity, toward the original normal color, toward real color, as a pre-processing, before recognizing the face, by face recognition module, to increase the accuracy of such recognition. The adjustment is based on the environment and background, so that color and intensity of pixels of the face is changed or corrected, such that the background becomes corrected to the normal or expected background, as if it were with/under enough light
Contrast Correction or Adjustment:
Let's assume that the intensity of a specific pixel P1 is I1. If P1 is in the middle of patch of low intensity pixels I2 (as the first environment), it (i.e. the apparent intensity, I12) looks much brighter than to the average human eye, compared or with respect to the situation when P1 is in the middle of patch or region of high intensity pixels I3 (as the second environment), where It looks darker, with low intensity (i.e. the apparent intensity, I13), to the human eye. That is, the perception of intensity, for recognition, by human eye, is dependent on background or context or contrast to the surroundings, Mathematically, it means that, for intensity, for human perception:
I13<I12
Now, the machine (measuring the exact intensity) does not make such a mistake, and measures the exact intensity, regardless of contrast to the surroundings. So, to normalize the machine or real measurements with human perception, to adjust for such perception difference, one has to adjust for the ratio (IR) of (I13/I12), between the given image in the first and the second environments (2 different environments). Thus, the number IR is our correction factor. So, starting from real intensity measurements, to go to the human domain or perception, one has to adjust the intensity by IR as our correction factor, to get the perception values or apparent values, relevant to the human perception. To go in the reverse direction, i.e. from human perception to the real intensity values or measurements, one does the correction or modification based on the inverse of value IR or (1/IR). After the adjustment, or pre-processing, the face recognition or any recognition is performed, resulting in better accuracy and reliability for recognitions.
Searching and Extracting Information from the Images or Other Data, Using Z-Web:
In one embodiment, for pictures or images from the Internet, or other data, we find e.g., the web site by search bot or robot, and then extract the relevant information and tag them or put a summary of that image or web site or list the extracted information in a database or put the relationships and relevance and reliability factors and other Z-factors mentioned above) into our Z-web or on our own server(s) or computer network or server farm (called Qstore storage or module or computer or server). Now, a third party user can look at our Z-web, or other information mentioned above and stored on our Qstore, to use or extract or search or download those data, for a fee or for free, based on different business models, such as ad revenue on our web site.
Basically, in one embodiment, the data extracted and collected and aggregated by us for our Z-web or our Qstore, based on an image on a web site (as an example), is sitting as an extra layer on top of the web site, so that the user can access and get more information from the web site, through our Z-web or our Qstore. There are many ways to do this process. In one embodiment, the user U is at his PC (or mobile phone or device), with a browser, which goes to a web site Wsite and is interested in a data Dsite on Wsite, e.g., an image or text data or tel. number. Since Wsite was previously scanned by search bot, and all the relevant information regarding Dsite was extracted, analyzed, and stored in our Qstore (e.g., in a remote location), then the user L can manually go to Qstore to get more information about Dsite, as one embodiment. In one embodiment, the user automatically goes to Qstore to get more information about Dsite. In one embodiment, the user optionally goes to Qstore to get more information about Dsite.
In one embodiment, the information stored in Qstore is also stored in Wsite. In one embodiment, the information stored in Qstore is instead stored in or moved to Wsite, as an extra layer or shell or attachment or tag-along file. In one embodiment, the information stored in Qstore is also stored in multiple places for easier or faster access, e.g., server farm or mirror server or backup server or redundant server, e.g., in another location. In one embodiment, the information stored in Qstore has an expiration date, after which the information extracted from or related to Dsite is updated or re-extracted. In one embodiment, the network including Wsite is the Internet. In one embodiment, the network is a private network. In one embodiment, the user can e.g., do a search or query and look for some object on Internet, using a plug-in and a browser, to go to the web site Wsite, and then from that web site, go to our database or Z-web or Qstore, to get the information extracted from the web site, automatically. Alternatively, the user can go directly to Qstore, using a plug-in and a browser, to get the information extracted from the target web site Wsite.
In one embodiment, the process above is done with no plug-in. In one embodiment, the whole process is done automatically. In one embodiment, the whole process is done with the input from the user, or partially by user, or optionally chosen by user. In one embodiment, when the mouse is over an object or hover over it, the whole process is initiated automatically, e.g., a picture in a web site or name in a text is selected (e.g., by mouse or pointer or user's finger on touch screen, or on monitor or display or pad or input pad or device, or hovered over by finger or mouse without touching or touching, or by magnetic proximity or heat proximity from body, or capacitance changes or by electrical resistivity changes or pressure or piezoelectric changes, or RFID tag proximity, or image of finger recognition or fingerprint recognition or biometrics validation, or car key holder or ring proximity, or finger gesture or face gesture recognition, or finger stroke or sign recognition, or series of finger strokes pattern recognition). Then, the relevant information is obtained from Qstore about that text or image, and automatically shown or presented to the user, which is very convenient and useful for the user on Internet or any private network.
In one embodiment, the web site Wsite, can also request, generally or per usage, to have the information on Qstore be also displayed on their web sites, or added or stored or tagged or linked or shown in their menus, based on another plug-in or code or predetermined arrangement with Qstore for direct usage of their users or visitors. So, it would be a value added for them (Wsite), for convenience of their users or visitors. Thus, it would be a source of income for the operator of the Qstore, as a service to Wsite or licensing the software or increased traffic for Wsite, e.g., for ad money or income, to benefit the operator of Wsite, e.g., as the client or customer for Qstore operation, e.g., as its business model. In one embodiment, the information from Qstore is supplied to the user directly, e.g. for mobile users or phone owners, per usage or per month or per subscription, for a separate business model or income source. In one embodiment, due to the value of the information from Qstore, the Qstore, itself, can have its own web site and direct visitors, for its own ad revenue, traffic, and referral income. In one embodiment, the web site includes text, image, tel. numbers, links, video, voice, music, and the like. See e.g.,
In one embodiment, an informational/graphical reader or renderer process (e.g., a web browser or a software application to view files or content such as a PDF reader or a word processor) runs on a device (e.g., a user device) that takes the content deliver from network (e.g., from a web server, file server, document or content server, web service, or an on-line application running on Cloud or distributed network of servers). In one embodiment, the reader/renderer process receives data (e.g., Z-web data for the annotation of an image identifying people on the image) related to a resource (e.g., the image) referenced or provided by the delivered content, based on (e.g., an automatic) query from the reader/renderer process (or a plug-in or another process running on the user device) to QStore related to (e.g., identifying) the resource (e.g., by its URL, identification or location within content or document, and/or metadata such as date). In one embodiment, the reader/renderer process modifies/overrides/supplements the display/play back or presentation of the resource (e.g., on the user's device), by using the received data (e.g., from QStore) including the user interface interaction (e.g., by creating links and displaying annotations on the image). In one embodiment, further user interaction with the modified user interface based on the received data, invokes further queries to QStore to fetch more data about the item selected (e.g., information about the person so annotated in the image). An embodiment makes the content (such as images) whether in web page or a document link to other knowledgebase entities by fetching the content in an automatic search (e.g., by hots or background processes), analyzing the content within a context and/or by using feature detectors/classifiers, importing the features of the content into Z-web, using the knowledgebase to automatically annotate the content and associate such annotation with the content (for a later search), e.g., via indexing.
In one embodiment, the network entity delivering the content does not include a reference to QStore (e.g., resources, API, or query) embedded with its delivery content to the reader/renderer, and a query (e.g., automatic) is initiated be a process in the user's device (e.g., reader/renderer process) to fetch data related to the resources in the delivered content. In one embodiment, the network entity (e.g., a web site, Wsite) has the content embedded with resources, API, query, or tags referencing QStore and the renderer/reader uses such embedded resources to fetch data from QStore or to display/playback the content (e.g., included the use of scripts such as Javascripts).
In one embodiment, the reader/renderer sends information to QStore or a server, when for example, the user enters annotation on a resource such as a portion of the image. In one embodiment, the information is tagged with the user's ID (e.g., is logged in). In one embodiment, the sent information is queued for analyzer to incorporate into Z-web. In one embodiment, the plug-in provides the user interface to enter/edit annotations on the user's device. In one embodiment, a local service or process running on the user's device provide a local QStore or Z-web on the user's device, e.g., giving local access to the user's auto-annotated photo albums, using other database (e.g., email or contact) to automatically build the relationship links between people appearing in the photos and appearing in the email to/cc lists. In one embodiment, the local QStore or Z-web may be synchronized with those on the network (or Cloud). See e.g.,
Partial Matching:
In one of our embodiments, we have a partial matching on objects hidden or covered behind others, or partial understanding or recognition of patterns hidden or covered by other objects, or not fully visible for any other reason, such as bad or dirty or foggy lens on camera. We compare the partial pattern or image of the first object to the library of all possible objects in that setting or environment, for partial match, with assigned reliability, based on the estimated percentage of the visible part of the first object, to put or incorporate it in the Z-web, where the recognition is enhanced based on the multiple inputs from other sources to cross-verify and cross-recognize, as described elsewhere in this disclosure, even using partial recognitions with not full reliability, per object, or node on Z-web.
Here, we give an example for partial matching for image, but this method can be used for recognition or verification of text, sound piece, series of music notes, signature, fingerprint, face, or any other feature or object or pattern, that is partially lost, obscured, hidden, erased, or not detectable/visible.
In one example, we have the first object being partially-matching with n objects in our target library (e.g., T)1, T)2, . . . , TOn), with different overall reliability factors, RF1, RF2, . . . , RFn, respectively, for the full match. For example, part of the first object matches with part of n objects in our target library. For example, a “handle” (or an object which looks like a handle), as a part of the first object, may be a part of (a handle for) a kettle, as first target, or part of (a handle for) a bottle, as a second target. First, we determine how much the handle of the first object matches the handle of the kettle, and matches the handle of the bottle, and so on, as denoted by MO1, MO2, . . . , MOn, respectively (for example, using matching or recognition confidence or score). Then, we determine the percentage of size or importance or contribution or dimension or ratio of a handle with respect to kettle, and with respect to bottle, and so on, as denoted by PO1, PO2, . . . , POn, respectively (for example, using the ratio of the sizes or dimensions or number of pixels).
Now, in one embodiment, the overall reliability factors, RF1, RF2, . . . , RFn, for the full match, is based on (PO1MO1), (PO2MO2), . . . , (POnMOn), respectively. (In one embodiment, the relationship can be more general, i.e. as a function of those values (Fr), or mitten in terms of: Ff (PO1, MO1), Ff (PO2, MO2), . . . , Ff (POn, MOn), respectively.)
So, the maximum or optimum reliability factor corresponds to (as a Maximum function, for taking the Max values on multiple parameters):
Max ((PO1MO1), (PO2M)2), . . . , (POnMOn))=(POkMOk)
Let's assume that the Max function above yields (POkMOk), as the k-th term in the series above. That is:
Max ((PO1MO1), (PO2MO2), . . . , (POnMOn))=(POkMOk)
Thus, the k-th object is the best target object for the full match.
Now, in addition, we can construct the relationships, to put all n objects in our target library into the Z-web, as described elsewhere in this disclosure, to find or recognize the best target object(s).
In one example, the problem is generally false positives, for recognition of target objects, but in one embodiment, with keeping track of reliability in our Z-web, we can tame that false positive rate to a reasonable quantity, making Z-web an extremely useful tool and technique for this type of situations.
Tags and Comments for Pictures and Images:
Picture annotation and caption is useful for recognition of people in the image, e.g., looking for phrases such as “from left to right”, or “top row”, to find location of faces or people in the image, and order them in rows or columns, and then call or label them as objects or persons PR1, PR2, . . . , PRN, as placeholders for names, and then compare them with the names coming after the flagged phrases such as “from left to right”, to get names matched with placeholders PR1, PR2, . . . , PRN. For recognition of names and flagged or pre-designated phrases, we use OCR and then basic or full natural language processor module.
In one embodiment, we can simply look for specific words such as “left”, as flagged words, and if successful, then look for specific phrases, such as “from left to right”, as flagged phrases, from our library of flagged phrases and words, pre-recorded and stored, or dynamically adjusted and improved through time, without actually understanding the meaning of the full text and sentence, for fast picture analysis and matching names or tags or comments related to the pictures.
In one embodiment, we can ask the user or third party, e.g., friend or public, to tag names or objects, or as crowd-sourcing effort or by voting scheme, e.g., paid service or free, or they do it on their own, because e.g., the (assuming unbiased) people familiar with a person may be the best or most reliable people to tag the album or pictures of that person, as an example. In one embodiment, the indicators can be used for approval, confirmation, or increase of reliability factor, such as “Like” for a picture or comment on FACEBOOK®, as an indicator of approval by a friend or third party. In one embodiment, the voting or survey is used for measuring approvals. In one embodiment, the comments after a video or picture is used, e.g., as overall positive or negative, e.g., “Great picture!” indicates approval and confirmation of a third party.
In one embodiment, the number of comments, number of views of a video, minutes watched for a video, length of comments, frequency of comments, date of recent comments, number of independent commentators, traffic of a web site, number of independent visitors to a site, number of followers on TWITTER® or other sites, number of connections, number of links, size of linked sites, quality of linked sites as rated by a third party, e.g., family-approved sites, number or size of advertisem*nts or advertisers, marketing budget, income, revenue, number of cited references by other sites or parties, e.g., for a research paper or patent or case law, or the like, might be indications for approval or reliability of source e.g., news, e.g., CNN-TV channel.
In one embodiment, the system automatically tags the pictures, and in another embodiment, it asks the user for verification. In one embodiment, it searches for a person in the album and sort based on that person(s).
Images from Different Angles or Views:
For example, we take pictures of the Eiffel tower from different angles, for training purposes, and store them, e.g., from top view and side view or from underneath. Some of the views are not common, and thus, unfamiliar to average human or eye. For example, if a picture of the Eiffel tower is taken from an airplane from exactly the top, the shape from the top may look like a square inside a bigger square, which does not look the same as a regular tower at all (or our average expectation or view of the tower). Various views help the recognition of the tower or object, as they can correlate or refer to the same object, which increases the reliability factor of the recognition or the recognized object.
In one example, given a picture, which includes a square inside another bigger square, the picture may also resemble another 2nd object, other than the Eiffel tower, in our library of the objects in the universe or at the specific location or city. Thus, other input information in the Z-web is used to increase the reliability of the data, and recognize the object, e.g., text or voice associated with the image.
In one example, given a picture, which includes a square inside another bigger square, one has to find the orientation of the image, from some templates in the library, or from thousands of training samples of many objects tagged versus direction and view by human or expert. The images in library can be real pictures or computer generated or drawn models, which compares shapes with each other, to find the best match, which indicates the view, e.g., “view from the top”. Once the direction or perspective of the view is determined, we can store that information into Z-web, to integrate with the rest of the information about the tower or object.
Pixel Patterns, as Feature Vectors:
For an image, we define the square cells, e.g., 32×32 pixels or 8×8 pixels. Generally, each pixel has 8 neighbors, such as top-left, top-straight, and so on. We start from one neighbor and go around all neighbors, e.g., in the clockwise direction. We compare the center pixel with each neighbor. If the difference of the center value minus a neighbor value is above a threshold, e.g., 30 points in pixel value, or above a relative size, e.g., above 35 percent, then we put “1” for that position. Otherwise, we put “0” for that position.
In another embodiment, we can do this with bigger range of assignment, instead of assigning only 0 and 1. For example, we can use 0 to 3 (or 0 to 7 range), to classify for finer differences, for difference between the center pixel and the neighbor pixel. Of course, we have a bigger overhead in this case, for computation power needed and for storage.
In either case, we end up with a cell with a bunch of numbers assigned for each pixel. These numbers indicate the local pattern of differences between neighboring pixels. In another embodiment, we can represent those assigned numbers in binary format for easier comparisons, as comparing 0 and 1 per position in digit order is very simple, using e.g., XOR logical operation. Now, we can use a histogram for over the cell, for the frequency of each assigned number in the cell, as an indication of the frequency of that difference or pattern in the cell, and also in the whole image. In one embodiment, we can normalize the histogram, for comparison of different histograms, based on the average values or median values or based on the ratio to the maximum value, i.e. ending up with fractions less than 1 for all values, which is more computing intensive. The histogram for all cells is an indication of the pattern in the image or feature vector, e.g., bar code black and white lines, or patterns or checkered or striped shirt or tie or fabric.
Now, the support vector machine and other classification methods can be used to classify the patterns or recognize the patterns or textures, such as for face or fingerprint recognition. The face recognition, as an example, can have multiple target people for comparison in the database of faces. If the new face is matched with one of the faces in the database, nothing new is created in the database. Only, the new face is tagged along or referenced, e.g., with a pointer, with the matched face in the database, as the variation of the person's face already in the database. However, if there is no match, a new account is created for a new person. If there is no name yet available, we add it under NO NAME category, using generic names of NONAME1, NONAME2, NONANE3, and so on, until we find a name match later on, which replaces the placeholder in every instance. For example, “John Smith” replaces NONAME3 in our Z-web configuration. Placeholder is also useful in the Z-web for names with low reliability, as “John Smith” does not replace NONAME3 in our Z-web, in this example. Instead, it creates another node, as a property of NONAME3, as a new node connected to NONAME3 node, with the value of assigned “John Smith” for the new node.
In one embodiment, we classify the faces already in Nface categories, e.g., 105 major types, based on regions of the world or shapes of faces, as a first level coarse classifier, so that the second level is a finer classifier to find a person. Or, if the number of faces in target database is huge, then we may need a third super-fine classifier, or even more levels of hierarchy for classifiers, feeding each other in multiple levels, for more efficient and faster classifications. In one embodiment, a human or expert or an already learned machine helps the training of a set.
Rule Templates Database:
In one embodiment, we have an image and we extract multiple objects from it, e.g., table and bottle, in which part of table is hidden or covered by bottle, which means table being behind bottle, referring to the positional situation of (from our position or relative location library): “bottle on the table”, or in general, object A located on object B, or also, meaning that object A closer to camera position than object B, which are stored in our Z-web. Then, later on, one can search or query about the position of the Objects and their relative locations, to extract these relationships. One advantage is that in Z-web, if object A is behind B, and B is behind C, then on the relational position between objects, one can conclude that A is probably behind C, for which such a template of rules are stored to support the Z-web, to help relate objects or simplify relationships, with the rule in a database of rules for Z-web, under category for positions of objects. Mathematically, the rule can be written as, where the function BE is the “Behind” function or operator:
If [[BE (B)=A] & [BE (C)=B]]
Then [BE (C)=A]
In general, the other logical relationships can be stored the same way in Rule Database engine (library), such as for “time”, or “over”, or “more”, or “before”, or “stronger”, or the like. For example, for “time” operator, if time A is before time B, and time B is before C, then A is before C. This can also be written similar to “Behind” function, in mathematical form, for template, for Rule Database. If the template is very similar for time and space, one can use a single super-template, as generic template, for both situations, to reduce the number of templates and increase efficiency, in some embodiment. See e.g.,
Rule database and templates can also have their own Z-web, relating the concepts, logic, relationships, and formulas, which can simplify the templates or get rid of the contradictions or inconsistencies. As an example, if we are not sure about a formula, we can store that as a rule in rule database, as a node with low reliability, which can be fixed, modified, or eliminated later on, on the rule database Z-web, which can be handled separately from our original Z-web structure. Alternatively, the 2 Z-webs can be combined as one super-Z-web, as explained elsewhere in this disclosure, with a common node being the object under study, such as “time”.
Image Analysis:
In one embodiment, architectural building signature is an indication of a region or culture, e.g., mosque arches in the Middle East, or white buildings near beach on the cliff, as Mediterranean style near Greek islands. The databases of famous people, pictures, paintings, locations, historical buildings, monuments, books, authors, architecture of cities and locations, and the like are incorporated with our analytics engine. In one embodiment, using OCR, we can extract the name of the book on the bookshelf in the picture from the library or book store, or name of the store, or name of the street, or name of the person on the door, or name on the business card, to find the person, address, business, or taste, or correlate them together, as some examples.
In one embodiment, the facts may dictate some limitations in the universe of possibilities. For example, the “snow” in “July” may indicate that we are in the Southern Hemisphere (of planet Earth), or the picture was taken from such a location, limiting all possible locations on the planet for candidate for picture location. See e.g.,
In one embodiment, travel guide is a good source of data for geography or history or facts. In one embodiment, the picture of an article either relates to the author or the subject of article. So, the face or person or author's name from caption or article or title or footnote should be extracted for comparison and classification or recognition of the picture or image. In one embodiment, the picture of an article in a web site is just an advertisem*nt, i.e., nothing to do with the article itself. In that case, we have to figure out that it is an advertisem*nt, from the caption or from subject matter or title or position on the web page or frequency of updates or functionality of the image. So, we have to partition the web page accordingly. In one embodiment, the GPS data or location data or time data or metadata, associated with a picture in a phone or camera, are used for data for Z-web for that picture.
In one embodiment, wax museum or movie setting is an indication of non-real people, even if the person looks like a famous people in the database. In one embodiment, a picture in a picture is analyzed, e.g., a picture in a movie or video frame, whereas the movie frame represents live humans in 3-D, but the picture frame on the table in the movie represents a 2-D image of a picture of a person, not a real person in the movie or video. Because to analyze the video, the 2-D image and the 3-D image may have different consequences and interpretations, e.g., as to who is related to who in the video. The 2-D picture frame on the table has specific signatures, e.g., it does not move around with respect to the other objects in the video, and has a constant coordinate.
In one embodiment, we have a database of famous people or US Presidents, e.g., George Washington, and database of famous places, e.g., Mount Vernon Estate, relating the two subjects or objects, as one being home of the other object. So, if we get a recognition of one object, automatically, the system looks for the other object in the vicinity, in terms of text or location or time or related objects or concepts, as expectation for other objects) to be around. That also helps confirmation of validity of the recognition. That also helps building up reliability factors for the Z-web structure, and expanding the Z-web.
Street Scanners:
In one embodiment, we have satellite or aerial images from buildings and streets, and if a new building is shown in a new image from this year, compared to last year's photo, then we send the “street car 3-D photographer street scanner” back in that street, to scan the new building and scene, as an update (using multiple cameras from different angles and views, on the car, plus scanner, on a rotating table, with GPS or location determination module, plus calibration images or data, to adjust for coordinates and views, with redundancies on images or data, to glue pictures together seamlessly, and to correct the fringes in images or corners, or to correct mistakes in images or coordinates or 3D views). If no new feature or building is detected, no new update on street level for street view is needed for the city map on the computer or web site. So, we are looking for features or deltas or differences, with respect to last year's or previous picture(s). Thus, we compute the difference between 2 images, from this year compared to last year, e.g., using simple difference or subtraction of values, pixel by pixel.
In one embodiment, from the amount of the differences in images, the system determines how often or when next time the street scan is needed or proper, to be dispatched or scheduled, based on importance of the city map for the users, and being up-to-date as much as possible or to what degree, in terms of financial value for the users or advertisers or local businesses or city hall or tourists or residents. If they have a fleet of those scanning cars, then the schedule is set to optimize the usage of those cars in different neighborhoods or cities, so that they get the best value for the users, based on differences in images in terms of amount and its frequency, user base and value per city or neighborhood, cost of operation of those scanning cars, and distances between the neighborhoods, to get most coverage and value, with minimum cost or mileage on the cars, or smaller number of scanning cars used.
Camera Corrections:
In one embodiment, the lens of the camera is scratched or dirty (e.g., with dust on the lens or oily lens, diffracting the light) or defocused or otherwise foggy and degraded (e.g., as if the transformation of Fimage (x) is applied to each pixel). Then, the picture does not come out very well, and the recognition of murky objects in the image is very difficult and with a high error rate. So, we filter the image, first, as a pre-process, to focus the image, as the reverse of the lens problem, as a reverse transformation on the image, or Fimage−1((x), applied to each pixel, to produce the corrected image. Then, we perform the recognition step, on the sharper or clearer objects or images, for improved recognition rate.
In one embodiment, for a camera taking pictures of an object, we have a family of transformations of Fimage (x) on the image or pixels, separately designed for each of these situations, to mimic the situation or effect on pixels or image: e.g., camera shaking, camera tripod shaking, object shaking, object moving in linear fashion, object rotating, blurred lens, dirty lens, scratched lens, oily lens, defocused lens (e.g., too fax or too short for focal length), off-axis lens (e.g. astigmatism or refractive error of the lens), dust on the lens of camera, and the like, which are the common reasons for blurry or degraded or defocused pictures by a camera. All the family of transformations Fimage (x) are stored in a library or database, for future access. The transformations Fimage (x) are designed or derived based on the optics or physics of the lens or theoretical formulation or analytical or pure experimental or simulation or optics model or physical model or pure curve or pure listing or table or closed form formulation or equation or combination of the above or the like.
Then, for each of these transformations Fimage (x), we derive reverse transformation on the image, or Fimage−1 (x), applied to each pixel or image, analytically, experimentally, theoretically, in-closed-form, by mapping numbers, by table of numbers, by simulation, or the like. Since we may not know the cause of the problem, or even if there is any problem in the first place, in a given image, we try all or some of the family of reverse transformation (Fimage−1 (x)) on all images, or on blurry images with bad recognition rate, or only on one or few sample images, to see if the recognition (e.g., recognition rate or reliability, e.g., on some samples) is improved or the blurring is reduced (e.g., based on sharpness of lines or borders). If so, then we know what the problem was for the camera, and we use that specific reverse transformation for all images from that specific camera or lens. If there are 2 or more problems e.g., with the camera, then we need 2 or more (e.g., N) corresponding reverse transformations (F1image−1 (x), F2image−1 (x), F3image−1 (x), . . . , FNimage−1 (x)) on the images, applied to the images in the reverse order, to compensate for the problems e.g., with lens or camera. After the images are corrected, then the recognition steps are done, which yield improved results.
If we already know or guess what the problem(s) is, then we just try that corresponding specific reverse transformation FMimage−1 (x), first. For the improvements, on recognition or blurring, we can have a threshold or rule or criteria or fuzzy rule or rule engine, to stop the process at that point, if we reach the threshold. The threshold can be fuzzy value, or crisp number, or percentage, or relative value or ratio, or absolute number, or the like, as the criteria for optimization.
Let's look at one example. For a defocused image with a defocused lens, we have a situation that e.g., the proper image is not formed on the plane of the film or photosensitive detector. Let's assume that the proper image would have been formed on an imaginary plane behind the current actual plane for the film or photosensitive detector or sensor. Let's also assume, from the geometry of the imaginary plane and the actual plane, the distance between those 2 planes produces e.g. approximately 2 pixel shift, on the actual data, for the current actual plane, because the optical beams or rays or photons hit the actual plane sooner than they should have, due to the defocusing effect of the lens. Thus, in this example, for a pixel (i, j) on the actual plane, to get the corrected value for the pixel, VC (i, j), based on the original pixel values, V (i, j), we have approximately the following relationship, based on the neighboring pixel values, from 2 pixel away, e.g., in one embodiment, on each of the 4 directions, e.g., up, down, left, and right sides, having 4 component contributions, as the sum of all 4 contributions:
VC(i,j)=V((i+2), (j+2))+V ((i+2), (j−2))+V ((i−2), (j+2))+V ((i−2), (j−2))
To normalize, we get the average of 4 contributors above, by dividing by 4:
VC(i, j)=[V ((i+2), (j+2))+V ((i+2), (j−2))+V ((i−2), (j+2))+V ((i−2), (j−2))]/4
Or, in another embodiment, we use 8 directions, including the diagonal directions, for neighboring pixels, with 8 component contributions. The method above for calculating the values can be applied to the intensity values, or each color component values, or each property value of pixel, e.g., RGB values or YMCK values or grayscale values. Now, we have the corrected values for pixels which correspond to the inverse transformation mentioned above.
The formulation above applies to all pixels in rows and columns, for all values of i and j. So, we have to scan the image. However, for pixels near the corner or boundaries, which do not have e.g., any neighboring pixel to the top or left, then we repeat the same value again for missing pixels, so that the formula above is still applicable. In summary, at the end, we can correct the image, to reduce or eliminate the defocusing effect, and then apply the recognition module on the corrected image for better recognition.
In one embodiment, we use convolution with a radial function, e.g., Gaussian function, with the variance of e.g., 2-4 pixels (or morel,and move it around, to scan the whole image, to get the same effect as above.
Geometrical Analysis:
In one embodiment, hom*ography and projective transformation can be used to compute camera rotation or translation, to account for a new point of view for a person or object, e.g., to match 2 views of the same face, from front and side, from 2 pictures, to match faces or recognize them.
In one embodiment, using filters for sharpening the edges as preprocessing, and then using contrast analyzer, between values of neighboring pixels, as their absolute or relative difference, versus a threshold or percentage, one can find the boundaries of objects (or using any other boundary analyzer). From the boundaries, one can find the corners of the objects, as their intersection of 2 boundary lines, or as the points in which the derivatives or slopes of boundary lines or curves change too much or abruptly or above some threshold. Corner points or boundaries are categorized as interesting points for the purpose of the feature extraction form the image, which collectively make up a feature vector in our feature space. Also, having all the corner points, the shape of the object can be found or named, from the geometrical shapes in the database.
Sorting & Analyzing Data:
In one embodiment, having “big data” coming in as input, we distinguish images in the first cut, very coarsely, e.g., text, medical images, satellite images, human faces, numbers, tables, computer codes, and the like, from their respective signatures and features, in training schemes or against databases already tagged. One example is text in different languages, as a sub-category, in later filtering or narrowing the class further, or typical street maps, which can be trained or learned using mil lions of samples, from that class or subclass. The learning machine generally works better with more training samples, as long as the samples are reliable (e.g., with high reliability factor, which can be extracted from their corresponding Z-web values).
In one embodiment, when analyzing a Big Data, the system comes up with or extracts some patterns or relationships at the beginning. Then, we store the patterns or relationships as templates for future use. As the time passes, the number of generated templates increases, increasing the value of the library of templates, and increasing the choices and possibilities for templates to fit in. Thus, at the end, we have some templates, from history, as educated guesses. For example, we can offer this as a service on the cloud, with all the templates generated so far, to analyze the data. In one embodiment, we export the templates from another system, into the first system, to add value to the template library. In one embodiment, our system buys or sells the templates from/to another system or source or entity. In one embodiment, the system uses the templates to analyze the data or extract information or data mine the data.
The examples of Big Data or data analytics are on the following data types: unstructured data, structured data, machine generated data, tables, listings, databases, collections, records, financial history, employment history, resume, business process logs, audit logs (file or database), packet data, industrial control system data, network state or status data, web proxy logs, system events, applications logs, click information (e.g., on Internet, web pages, buttons, menus, objects, figures, and the like), database logs, logging API, operating system status, information obtained from sensors or meters or detectors or cameras, web access or network access logs, texting records, SMS records, call records, TWITTER.® records, configuration files, management API, message queue, operating system performances, data from control and data acquisition module, satellite images, input from airport cameras, movie scans, music scans, speech scans, text scans, medical images, library scans, database scans, or the like.
The analysis of the above data e.g., can be used for predicting customer behavior, finding correlations among sources, forecasting sales, catching fraud, finding computer security risks, processing sensor data, social network analysis, feedback analysis, emotion analysis, web click streams analysis, or the like.
Recognizing Objects for Various Applications:
In one embodiment, we identify people in the picture in album or by camera or video recorder, and automatically as default (unless chosen otherwise from the menu), email to all people involved or recognized through the album or from camera or from each picture, from their contact list, if desired, or to all in the event, or share through link or FACEBOOK®. The scope of distribution is set beforehand, for list of recipients. For example, if three people are in one picture, namely John, Amy, and Fred, then that picture is emailed to those 3 people, only. However, the next picture has e.g., 4 people in it, namely, Ted, John, Amy, and Fred, and the next picture goes to all 4 people, including Ted, and so on. The preferences are chosen beforehand for templates or single picture or batch processing, for all or subset of pictures or data or video.
For example, there are 2 people recognized, out of 10 people in the video, from our library. The other 8 people were not in our library or recognition was not successful. Then, a copy or link or track or frame number or pointer or position of the video or web site or storage for the video or specified frame of video is sent to the 2 friends that are recognized from the searched video. That can help for social networking sites and professional settings for a conference call between some co-workers, automating distribution of relevant data, including voice, text, video, or image, that include the name of specific people or image of the person or any related object to that person, found in that text or sound piece or video or image, to be sent to that person automatically. See e.g.,
This can be used for example for copyright or trademark protections, in which the image including a person's copyright or trademark is sent automatically to the person for review, out of millions of web site pages scanned on the Internet, to find infringers or verify licensee payments, if any. Or, one can send the list of web sites using a specific logo or trademark or patent to a comparison module, to verify against the list of legitimate or paid or permitted licensees in its database, automatically, to approve or warn the related people, by email, pre-recorded voice message, texting, SMS, mail, vibration warning on the phone (e.g., specific vibration or cycle or sequence or variable magnitude or variable frequency), any communication means, or the like. So, it is a useful tool for sending information to relevant people, automatically, by email or tel. (via text, voice, or image) or any other communication means, once the object is recognized in the content under review, and the object is linked to an entity that subscribes to our services, such as friends or corporate legal department, for fee or for free, depending on the business model or purpose of the service.
For example, if I am interested in movies by director A, then any new or old movies found for her name can be automatically sent to me, even if the mention was on text or verbal (sound/voice) at the end of the movie, with no tags or comments. Of course, any tag or comment on the movie regarding that information makes it easier for such a process. Alternatively, those tags and comments can be verified based on the other data extracted from the video directly by Z-web engine, e.g., text or sound naming the director at the end of the movie. For those, we use OCR or voice recognition modules to recognize and convert information for comparisons.
The picture or sound albums or videos can be classified and archived this way, in a cross classification way, using a relational database, for relating e.g., objects, videos, and people together, in a final Z-web structure, and to be searched by the user later on, as a query, or question, about any subject, starting from one node and going to other nodes, even if the user does not know that the second node is related to the first node at the beginning. This is also a powerful search engine and archive, which is expandable by its knowledge base through expanding Z-web size and making more reliable and consistent and truthful branches and relationships on the Z-web, increasing the total value of the Z-web. For example, the same picture or video or voice speech may be referenced in multiple places for multiple reasons for multiple objects or nodes or branches, which can be queried or searched independently, through Z-web.
In one embodiment, we identify Objects in the video or images for advertisem*nt purposes, or for consumer purposes, to send ad notices or notify the potential buyers or notify new consumers or notify about new products or requested types of products or products of interest. The object in video, for example, has some relationship with the recipient of the email or communication or texting or notice or telephone call or fax or ring tone or the like, as a way of notification. For example, the relationship came from the Z-web, or the object was chosen by the user, or the class of objects was chosen by the user or third party or randomly by computer, through the menu or the user interface or GUI or tablet screen or tel. screen or by voice recognition command. So, the extracted object can be the subject of an ad, or suggested product, or put into a cart for purchase on web site, or sent to a reviewer, or stored in a database, or broadcasted to many people, or the like.
One can also search using a query for the album, e.g., using a text, to find an object. One example is to find out that, for the identified person in video, what kind of food or drink does he like? Those could be found through the other objects in frames (nearby) in video, e.g., on or near the person's desk or in his hand, holding the drink or bottle. Or, if somebody enters a textual or voice question for the system, as what kind of food or drink he likes? Then, we can use OCR or voice recognition or analysis to get the question, and then use word search or natural language processing or specific flags for key words, to get the meaning of the question, or approximate meaning of that. Then, we apply the method mentioned above, to answer the question(s) or find an approximate answer.
In one embodiment, we identify an object in the video or images, then we remove or edit it, or replace it with another object, e.g., for advertisem*nt or localization purpose. For example, in different countries, different beer (local beer) is used in pictures or movies or ads, for better acceptance as a local favorite or for marketing and higher sales, by replacing one object on the table with another one from library for local objects, e,g. beer bottle, to be put on table, seamlessly and automatically. See e.g.,
In one embodiment, we recognize a partial object (1st object), which is behind another object (2nd object), and for editing purposes, in the image or for movie (for all frames including that object), we bring the full image of the 1st object in front of the 2nd object, to block the 2nd object partially, by the 1st object, in effect reversing the order of the objects in the still image or video frames, in any order we wish. The image of the 1st object is in our library of objects, which can be obtained from there. Then, the size or color or intensity is normalized to the one in the image, so that we do not feel any abrupt change in size or color or intensity, based on the ratio to the values in the neighboring pixels, or average in a region, or value of pixels near border lines or boundaries. For the perspective, if the 1″ object is tilted, then either we use the tilted version of the 1st object from library (if available), or we morph the image of the 1st object in the library slightly, based on translation, rotation, lens, or similar image transformation matrix, to look similar to the tilted image of the 1st object, to be replaced, for editing purposes.
In one embodiment, we recognize faces in an album, and find the incorrect tagged ones, or edit and correct them automatically, with or without the user's or owner's permission or review, as multiple options in the software.
In one embodiment, we recognize faces in the album and insert it automatically in the phone display or screen, when the person calls in, to be displayed, with a specific ring-tone, or mentioning the person's name, in voice or text, to inform the receiver regarding the identity of the caller. In addition, the mute vibration mode can have different frequency of vibrations, set for different users or callers, so that they can be recognized by a mix of vibrations at single or multiple frequencies, or using notes or music style beats or vibrations, or using modulated waveforms as vibrations lasting a few seconds, for example.
In one embodiment, we recognize faces in the album and insert it automatically in the email, for sender or receiver, so that it would be easier to recognize the people in the email list, and fewer mistakes will happen for sending an email to unwanted or unintended people. The reverse can also be done. That is, we select pictures from the album, and the email list is generated automatically, from person's identity, which relates to the person's contact information, e.g., email addresses or telephone or fax numbers, which all can be part of the person's Z-web, as the related information to the person's node, represented by neighboring nodes.
Data Extraction, Including Emotions and Taste:
In one embodiment, the signature of the cell phone or ID number for camera relates the picture to the person who took the pictures or owner of the camera, which relates the picture to the friends of owner, which relates the picture to possible targets for people in the picture(s), for recognition. In one embodiment, the pictures from nature or mountain or cities relates to the taste or preference of the owner of the camera or cell phone camera, which relates her to the possible destinations for the next trip, which is helpful for marketing for travel agencies or local ads for products or services for those destinations, such as local rental car companies. In one embodiment, the pictures from a house in camera are used for extracting the taste of the owner of the camera, for the house setting and environment, such as in suburb or wooded area, for future house hunting for the new home buyer (camera owner), which is helpful to the real estate agents, for the preferences of the home buyer, for marketing or efficient house hunting.
In one embodiment, “smiling” in a picture is used to find emotions for the people in the picture or the situation in the image, such as celebration and birthday ceremony, as opposed to sad situations such as funerals. In one embodiment, smiling is recognized using the big library of smiling pictures of real people for comparison or training samples. In one embodiment, smiling is recognized as a symbolic (caricature) basic shape of the mouth versus the general shape of the face, in relation (or with respect) to each other. For example, smiling is defined as a curved mouth with both ends going upward, or a strike similar to a “U”. That is, as long as we can distinguish such a relationship for the mouth with respect to the face, or such a general shape for the mouth, we can tag that as a smiling picture or person. This can be done for any emotions, such as angry, crying, shouting, and the like, for various contexts, for example, a sad situation, for example, for funeral, to relate the black dress and sad situation to family members in the picture or scene, for common loss of a family member, who is possibly the one of the few people missing in the scene or pictures, as extracted from the family tree or family album or family names tagged in the album or FACEBOOK® or similar social web sites, for all names in the whole family. See e.g.,
In one embodiment, the emotion is related to the character of the person, mood, intention, future action, state of mind, or psychology, e.g., one person being angry at some event may indicate his sad mood or his intention to spoil the event. These can be modeled through Z-web and Z-nodes.
Another example is when we have e.g., a total of 5 friends in the album for the trip or vacation to Miami Beach in 1995, which means that the universe of all buddies in that trip is 5 people, which is extracted as our system reviews all the pictures from that trip, with maximum 5 distinct faces recognized there, at the end of the evaluations. So, e.g., some pictures have 2 people and some have 4 people in them. For example, the missing person in a picture with e.g., 4 people in it might be the person who took that picture, and he might be the camera owner, as well, if most of the pictures are like that (i.e. him being missing from all or most of those pictures, in that camera).
In one embodiment, we find all objects in a picture and summarize them as data, templates, tags, comments, numbers, and the like, which can also be used for trainings for signatures or features of other images for future. In one example, we have about 5000 different objects in our library, for most often used objects in everyday life, such as shoe and door, which can be sub-classified for fast search and retrieval, such as office furniture. These objects are also represented in a Z-web, as related objects, e.g., computer and mouse.
In one embodiment, the type of beverage, wine, suit, car, fruit, clothing, cigar, and the like are also some examples of taste of a person. In one embodiment, when get some value for an object, then we instantiate all instants of the object with that value, e.g., object in a satellite image is a tank, or in a medical image is a cancer cell or tissue. Meanwhile, we can put a placeholder name for that object, until it is recognized.
In one embodiment, we do multiple steps hierarchy recognition, to get many images and analyze coarsely to put them in the right bins or classes (e.g., picture of people), as preprocessing, and then, go finer and finer analysis or filtering, to get into specific data, e.g., find or locate faces, and then face recognition. Another example is for recognition in different levels, e.g., starting from finding all radiology x-rays, then bone in image, then foot as the subject of the image, then broken foot as the property of the foot, or age or sex of the subject, from the parameters of the image, based on expected values in the medical databases, e.g., for our prior samples or training samples for neural networks.
In one embodiment, we have a face recognition based on the chunks or pieces of face, e.g. recognizing nose or lips, individually and with respect to each other, to confirm that they constitute a face, e.g., with respect to relative position or size. The parameters are all fuzzy parameters, in one embodiment. The relationship and relative position or size can be expressed through our Z-web, as a method of recognition of an object, with all its components, to first see that it is actually a face, and if so, whose face it belongs to, i.e. recognize the person in the next step. The shape and size of the components of a face or object are expressed in fuzzy relationships or fuzzy rules, in one embodiment. Or, it can be stored as a target object or training sample in a database or library or storage, for recognition, training, and comparison purposes.
In one embodiment, from a picture of food plate, the system extracts the objects and recognizes them, e.g., peanut, and from the library, the system gets all the nutritional facts, for proper diet for the user, per day, as accumulated and compared with the special or recommended regimen, for general or for a specific person or patient, to limit or warn the user or to recommend or remind a user, e.g., for deficiency of calcium or for allergy to an ingredient or for conflict between foods and drugs, stored in library for general knowledge and also on a separate database for a specific person, as customized, on her laptop or smart phone or mobile device. In one embodiment, such information is integrated into the routine for exercise for the same user, for scheduling and count and statistics and progress report. See e.g.,
In one embodiment, for any picture that does not come out right (e.g., no smile, rotated head, or closed eyes), the system tags the pictures for review by the user, or in one embodiment, optionally, the system automatically deletes them from the photo album or frames, or exchanges them with the approved ones or good pictures or neighboring pictures or similar pictures, or leaves them as blank.
Cost of Search:
The cost of search in terms of computational power and delay time is a factor, as how far we want to go deep to get to other related nodes to find other related objects for more complete search, in our Z-web. For example, as one detects a computer mouse, then the system looks for a computer nearby, within the expected radius of proximity between 2 given objects, which is stored in a database or in a Z-web, as a parameter shared between computer and mouse nodes, as explained elsewhere in this disclosure. Thus, for a given computing power and time, one can estimate how deep and wide the search for related nodes is, and what and how many related Objects can be obtained or analyzed.
The search or traversing the nodes can be directional or biased intentionally, for example, for one embodiment, for a geographical issue, one may expect more geographical or location related nodes. So, we follow the nodes that are more inclined or related to locations, such as “restaurant” or “coordinate values of a location on GPS or planet Earth”. The selection of branch can be optimized, to go deeper in one branch versus another one, in a Z-web structure, to find related objects or nodes. With a parallel processor, the selection of multiple branches can be done simultaneously.
Another Way of Calculating “Z-Factors”, Including Reliability Factor:
Please note the reliability factor can be calculated based on the other methods mentioned in this disclosure. This can be also calculated and addressed by Z-numbers or by fuzzy logic membership or other fuzzy concepts or other concepts, such as the “trustworthiness of speaker”, “sureness of speaker”, and “statement helpfulness”, which deal with the source of information, where the information propagates through one or more sources to get to the listener or user, as the final destination, to analyze the information and its quality', including reliability factor, confidence factor, truth factor, bias factor, expertise factor, validity factor, expiration date (if any, to declare the information void after a certain date and time, such as stock market quotation), and the like (collectively called Z-factors, for factors used in Z-web)
For example, for reliability factor analysis, in another embodiment, we have e.g., for 3 nodes N1, N2, and N3, where the information is moved from N1, to N2, and then to N3, with reliability factors RF1, RF2, and RF3, respectively. For example, assuming all being normalized to maximum the value of 1, then all RF1, RF2, and RF3 are less than or equal to 1 (or it can be done in the percentage scale to the maximum value of 100). So, in one embodiment, we will have the total reliability factor RFTtotal as the multiplication of all factors in the series of nodes:
FFTotal=RF1RF2RF3
In one embodiment, we will have the total reliability factor RFTotal as the intersection of all reliability factors in the series of nodes, or minimum of those values:
RFTotal=Min(RF1, RF2, RF3)
In one embodiment, we will have each reliability factor is compared to a threshold, in the first case as being larger than a threshold to get a value of 1, and in the second case as being smaller than another threshold to get a value of 0, which makes the calculations simpler for calculations of formulas above for large number of nodes, because we end up with lots of 0 and 1 in the node factor assignments.
For parallel nodes situation (as opposed to series), we will have the corresponding formulation for the total reliability factor RFTotal, for example, for 3 nodes (N1, N2, and N3), going to a final 4th node, N4, as parallel inputs. If the information coming from all 3 nodes (N1, N2, and N3) are not related, then they have no impact on each other in terms of reliability. However, if they are related to the same subject, then we will have, in one embodiment:
RFTotal=RF1+RF2+RF3
In one embodiment, we will have the total reliability factor RFTotal as the union of all reliability factors in the parallel configuration of nodes, or maximum of those values:
RFTotal=Max (RF1, RF2, RF3)
Again, in one embodiment, we will have each reliability factor is compared to a threshold, in the first case as being larger than a threshold to get a value of 1, and in the second case as being smaller than another threshold to get a value of 0, which makes the calculations simpler for calculations of formulas above for large number of nodes, because we end up with lots of 0 and 1 in the node factor assignments.
If we are dealing with fuzzy numbers, then we can use the operators max, MAX, min, MIN, and sup, as commonly known in Fuzzy Logic, and e.g., as defined and shown by
All of these methods in this disclosure can also apply to other factors mentioned elsewhere in this disclosure, e.g., confidence factor, truth factor, bias factor, expertise factor, trust factor, validity factor, “trustworthiness of speaker”, “sureness of speaker”, “statement helpfulness”, “expertise of speaker”, “speaker's truthfulness”, “perception of speaker (or source of information)”, “apparent confidence of speaker”, or “broadness of statement”. The mathematics and vehicle to apply to Z-web nodes (also called “Z-node”) or objects are the same for each of those factors (collectively called “Z-factors”, for factors used in “Z-web”). The collection or aggregation of Z-web with all the associated factors mentioned above makes it the most reliable and most powerful search engine tool in the market, for data analytics or analysis of images, “big data”, text, voice, moods, facial expressions, emotions, personality, character, characteristics, concepts, and the like. Of course, the bigger Z-web gets, the more valuable it becomes, with more nodes and factors and branches and other parameters, as mentioned above.
In one embodiment, “trustworthiness of speaker” (Atrust) depends on (as a function of, or Function(x)) at least 4 other factors (variables): “apparent confidence of speaker” (Aconfidence), “speaker's truthfulness” (Atruth), “expertise of speaker” (Aexpertise), and “perception of speaker (or source of information)” (Aperception), with each can be both fuzzy and crisp values, in different examples. In one embodiment, the “trustworthiness of speaker” is “high”, only if all of its 4 factors are “high”. So, mathematically, we have:
Atrust=Function (Aconfidence, Atruth, Aexpertise, Aperception)
If we assign the value of 1 to “high” and 0 to “low”. In one embodiment, then we can write this in a short form as, based on AND logical operation:
Atrust=(Aconfidence AND Atruth AND Aexpertise AND Aperception)
Or, in another presentation, in one embodiment, we can write it as, using intersection operator (̂):
Atrust=(AconfidencêAtrutĥAexpertisêAperception)
Or, in another presentation, in one embodiment, we can write it as, using minimum operators (e.g., min or MIN, as commonly known in Fuzzy Logic, and e.g., as defined and shown by
Atrust =min (Aconfidence, Atruth, Aexpertise, Aperception)
or
Atrust=MIN (Aconfidence, Atruth, Aexpertise, Aperception)
So, we can calculate or obtain Atrust from its components or variables, based on fuzzy rules, set rules, logical operations, Venn diagram, or the like, for their respective domains of analysis.
Note that for any intersection operator or logical or fuzzy operations, mentioned here, we can use different logic domains and operations, e.g., Lukasiewicz logics, Bochvar logics, Kleene logics, Heyting logics, Reichenbach logics, or the like (see e.g., Table 8.4 of Klir et al. (on page 218 of the book mentioned above)). In addition, for the Generalized Modus Pollens, Generalized Modus Toliens, and Generalized Hypothetical Syllogisms, we can use the following conventions, as an example: Early Zadeh, Gaines-Rescher, Godel, Goguen, Kleene-Dienes, Lukasiewicz Reichenbach, Willmott, Wu, or the like (see e.g., Tables 11.2, 11.3, and 11.4 of Klir et al. (on pages 315-317 of the book mentioned above)). In one embodiment, to be consistent, once we are using one logical domain, we have to stay in that domain for all operations.
In one embodiment, “sureness of speaker” (Asureness) depends on at least 4 other factors: “apparent confidence of speaker”, “speaker's truthfulness”, “expertise of speaker”, and “perception of speaker (or source of information)”, with each can be both fuzzy and crisp values, in different examples. In one embodiment, the “sureness of speaker” is “high”, only if “speaker's truthfulness” is either “high” or “low”, and the other 3 factors are “high”. So, mathematically, we have Asureness as a function of:
Asureness=Function (Aconfidence, Atruth, Aexpertise, Aperception)
If we assign the value of 1 to “high” and 0 to “low”. In one embodiment, then we can write this in a short form as, based on AND and OR logical operations:
Asureness=Aconfidence AND Aexpertise AND Aperception AND (Atruth OR Ãtruth)
Wherein Ãtruth is a logical complement to Atruth. In fuzzy logic, please note that, due to overlapping membership functions, (Atruth OR Ãtruth) is not equivalent to 1.
Or, in another presentation, in one embodiment, we can write it as, using intersection operator (̂) and union operator (V):
Asureness=AconfidencêAexpertisêAperception̂ (Atruth V Ãtruth)
Or, in another presentation, in one embodiment, we can write it as, using minimum and maximum operators (e.g., max, MAX, min, MIN, and sup):
Asureness=min (Aconfidence, Aexpertise, Aperception, (max (Atruth, Ãtruth)))
or
Asureness=MIN (Aconfidence, Aexpertise, Aperception, (MAX (Atruth, Ãtruth)))
Or, we can use any of the combinations of the similar operators, listed above. So, we can calculate or obtain Asureness from its components or variables, based on fuzzy rules, set rules, logical operations, Venn diagram, or the like, for their respective domains of analysis.
In one embodiment, “statement helpfulness” (Ahelpfulness) depends on at least 2 other factors:
“sureness of speaker” (Asureness) and “broadness of statement” (Abroadness), with each can be both fuzzy and crisp values, in different examples. In one embodiment, the “statement helpfulness” is “high”, only if “sureness of speaker” is “high” and “broadness of statement” is “low”. In one embodiment, “statement helpfulness” indicates the parameter that is very useful for analysis of many pieces of data from multiple sources, such as Big Data or Internet. So, mathematically, we have Ahelpfulness as a function of:
Ahelpfulness=Function (Asureness, Abroadness)
If we assign the value of 1 to “high” and 0 to “low”. In one embodiment, then we can write this in a short form as, based on AND logical operation:
Ahelpfulness=Asureness AND Ãbroadness
Wherein Ãbroadness is a logical complement to Abroadness. In fuzzy logic, please note that, due to overlapping membership functions, (Abroadness OR Ãbroadness) is not equivalent to 1.
Or, in another presentation, in one embodiment, we can write it as, using intersection operator (̂):
Ahelpfulness=AsurenesŝÃbroadness
Or, in another presentation, in one embodiment, we can write it as, using minimum and maximum operators (e.g., max, MAX, min, MIN, and sup):
Ahelpfulness=min (Asureness, Ãbroadness)
Or
Ahelpfulness=MIN (Asureness, Ãbroadness)
So, we can calculate or obtain Ahelpfulness from its components or variables, based on fuzzy rules, set rules, logical operations, Venn diagram, or the like, for their respective domains of analysis.
In one embodiment, the information comes from multiple sources or speakers (or originator or gatherer or reporter) and goes through more sources, and may get modified in there, based on the same parameters described above. Then, the information may get merged, edited, combined, aggregated, or modified by some sources, or otherwise, just goes through an intermediate source with no modifications, just as a conduit, with no effect on the data. Finally, one or more pieces of data reach a listener (or receiver or evaluator or user or computer or collector or public or third party entity), through those many possible routes (from one or more original sources of information). Then, the listener should gather all data, with all the factors mentioned above, from all sources and routes, and digest and evaluate, to make a conclusion from all of the above. Here, the Z-web is applied, because the whole propagation of data through all the nodes or sources can be modeled with the Z-web, from one part of the Z-web to another part or section or node of the Z-web, with all the reliability factors and other factors included in the Z-web. This is a very powerful tool for analytics e.g., for Big Data or Internet, with many sources of information, and many intermediate nodes, each having its own reliability, truthfulness, bias, expertise, addition, edit, and similar factors, e.g., as mentioned above, on the original data.
Of course, when we have a complex node structure for Z-web, we will have a multiple parallel and series situations, broken down as imaginary smaller units, which we can use the methods above or as explained elsewhere in this disclosure, to analyze for the Z-web. In addition to the above formulations, any similar formulations and combinations can also work in different embodiments. For example, the Z-factors can be based on tables, curves, formulas, analytical relationships, equations, Fuzzy rules, rules engine, conditional statements, or the like.
Processing & Mathematical Methods:
In one embodiment, root-mean-square error (RMSE) or ArcTan function (an inverse trigonometric function) can be used to reduce the effect of the large errors or dampen the big variations, because of their behaviors at large numbers.
In one embodiment, for optimization, the method of Lagrange multipliers is used for finding the local maxima and minima of a function subject to some constraints, which is equivalent to finding some optimum point on a surface, subject to a cross section of that surface, which is equivalent to tangent vectors of the 2 corresponding contour lines being parallel, which is equivalent to gradients of 2 functions (e.g., f(x,y) and g(x,y)) being parallel, subject to a constraint for g(x,y), e.g., being a constant value. So, we will have the following relationship for the gradients of the 2 functions (with the gradient being taken with respect to x and y (the 2 coordinate axes), and k representing a coefficient of proportionality):
∇f=−k∇g
We use this for any optimization, e.g., in our image processing method or recognition routines or image enhancements or optimization of one Z-factor in Z-web, subject to another Z-factor (as a constraint), e.g., reliability factor.
In one embodiment, for fuzzy system reasoning, for aggregation and implication techniques, e.g., we use Min/Max Aggregation method, in which we get the membership value from the 1st curve or membership function, and trim the top of the 2nd membership function or curve (as flat cut-off) with the first membership value (as the maximum value allowed on that 2nd membership function or curve). In one embodiment, we use Additive Aggregation method, in which we get the membership value from the curve or membership function, and scale or normalize the 2nd membership function or curve (usually reduce the peak height of the curve) based on the first membership value (as the maximum value or peak allowed on that 2nd membership function or curve).
In one embodiment, for aggregating the correlated fuzzy sets, for the additive aggregation method, we can get the final membership value, μfinal, based on the individual membership values, μ1 and μ2 , as (where index i runs from 0 to n):
μfinal=∀i min (μ1+μ2), 1)
In one embodiment, for aggregating the correlated fuzzy sets, for the Min-Max aggregation method, we can get the final membership value, μfinal, based on the individual membership values, μ1 and μ2, as (where index i runs from 0 to n):
μfinal=∀i max (μ1, μ2)
Thus, we can aggregate and find the resultant membership functions. Then, if we have to defuzzify at one point, to get a crisp number for some applications, e.g., acting or not acting on some function, e.g., turn off the light, as a binary decision or output, then for that case, we get the center of mass coordinate, or the coordinate of where the membership curve bends (going down from its maximum value or plateau or flat region on the top), or the coordinate of the center of the highest plateau or flat region of the membership curve (if any), or any averaging or weighted averaging or the final membership curve, or any computation for the area under the membership curve to find a compromise value or middle point or median point or coordinate, or the like. However, if the crisp value is not needed, we keep the values as fuzzy values, stored or sent to the next step of the process, because when we defuzzify, e.g., to get a point or crisp value, we usually lose some information stored in the membership curve. So, we want to avoid that, if possible, or as much as possible.
In one embodiment, for fuzzy rules, e.g., rules engine, e.g., for control system applications, e.g., when we have a bunch of rules in terms of e.g., IF-THEN rules, or conditional statements, then we can apply the rules or policies based on fuzzy backward chaining, to resolve the rules backward, to fire or activate the rule(s), in our inference engine. Here, we start from a goal, and then find out which rules have that goal as output, and from those extracted rules, we find out what input parameter values we need to know to evaluate those extracted rules. Those input parameter value(s) now become our sub-goal(s), which is (are) similar to our goal above, which we repeat the same routine above again, recursively, until we get to an input value that we already know the value for, and we can plug in the value for that corresponding rule, as the input, to get the output (of that IF-THEN rule). Then, recursively, we use that output that we just got as the input of the previous rule(s), concerning that parameter, in our chain, to get the result or output of the previous rule(s). Then, we continue this recursively, until we get to our first goal at the top of the chain, in the beginning.
In one embodiment, for fuzzy rules, we use forward chaining inference engine, to fire the rules in forward manner, as the input part of the rule has some available value, to trigger that rule, and instantiate some variables. So, we go through all rules that can be fired, until the list is exhausted. So, here, we do not require a goal.
In one embodiment, based on both approaches mentioned above, to take advantage of our gathered knowledge in each step, we combine both methods, as an opportunistic forward firing, added to our backward chaining engine, for better performance.
In one embodiment, we (i.e. our system or computer or processor or microprocessor or CPU or computing unit or the like) perform parallel processing for each signature for each object in the picture, to get the result faster, to extract and distinguish all objects in the image.
In one embodiment, we can combine our method with Adaptive Boosting, as a machine learning algorithm, designed by Yoav Freund and Robert Schapire, to improve the performance (AdaBoost algorithm). The subsequent classifiers are adjusted in favor of those instances misclassified by previous classifiers. In one embodiment, it is sensitive to noisy data and outliers. In one embodiment, it is less susceptible to the “overfitting” problem (which is a well-known problem in machine learning). The system generates and calls a new weak classifier in each cycle, with updated weights, based on the importance of examples in the data set. Then, repeatedly, the weights of incorrectly classified examples are increased, and those of correctly classified examples are decreased, to zoom in to the missed examples.
In one embodiment, we can combine our method with the following method, for classification, such as face recognition, to consider both the error and time for the final decision, based on sequential decision-making. When the false positive and false negative error rates are given to us, then we want the shortest average time to decision (number of measurements). We use a method called Wald's sequential probability ratio test (SPRT), or WaldBoost. We use both a priori ordered measurements and known joint probability density functions, for time and error rate trade-off, with the joint probability density estimation using the optimal SPRT decision strategy, which has a good detection rate in a reasonable amount of time.
In one embodiment, we can combine our method with the c-means clustering algorithm, which produces input pattern groups with corresponding cluster centers. To team fuzzy functions, one can use adaptive vector quantization (AVQ) (using unsupervised AVQ competitive learning) to estimate the local centroids (and covariance matrices) of clusters in the input-output space. From the resulting ellipsoid, one can derive the fuzzy rules and fuzzy patches). In one embodiment, one can use the Kohonen self-organizing map (SOM), with unsupervised learning algorithm, to change weight vectors for a network (for modeling the features in training samples).
In one embodiment, for thresholding in image processing, we use a statistical decision theory, using statistical decision rules. In one embodiment, we use Otsu's thresholding technique, which uses discriminant analysis, which maximizes the class separation. In one embodiment, we use fuzzy threshold algorithm, using fuzzy membership functions (by the method suggested by Huang and Wang). In one embodiment, the selection is made using a fuzzy parameter, with entropy measure as the criteria function, to get the threshold for the optimal image. In one embodiment, we introduce a cost function. In one embodiment, we apply the multi-level thresholding. In one embodiment, we apply a model fitting method. In one embodiment, we apply the above to segment the document images, face, text, or the like. In one embodiment, we use the grey level histogram for thresholding and segmentation purpose. The histogram (and its peaks or its transition phases) is a good indicator of the multiple classes or clusters involved in the samples.
In one embodiment, we use a fuzzy rule based system to find the background in the image. For example, we have the following IF-THEN rule(s), using Z-numbers:
-
- If, for a given pixel, the pixel's neighbors have small contrast and small variance with respect to the pixel, then the pixel is probably in the background of the image.
- Otherwise, the pixel is the foreground, representing one or more objects in the image.
In one embodiment, for learning from samples or examples, we have the following steps: First, we fuzzify the input space. Then, using data, we produce fuzzy rules. Then, for each rule, we assign a degree, followed by the creation of the combined rule library. Finally, we use defuzzification to set the mapping.
In one embodiment, for recognition, we use decision tree method, with nodes and branches which can represent the rules. For example, we have: “If D1 has a value of d13, and D2 has a value of d21 (on the second level of branching out), then the class is C2”, as shown in
If D1(d13) is TRUE, and Dd(d21) is TRUE,
Then the class is C2.
In one embodiment, we assume the decision tree classifies the objects with the same proportion as the samples in our universe of objects, i.e. with the same probability. In one embodiment, if we look at the decision as a source of message, then we can relate that to the entropy formulation for the information (I) (with summation running on variable “j”, and P denoting the probability):
I=−ΣjP(aj)log(P(aj))
In one embodiment, we use fuzzified decision rules based on membership functions, which have values between 0 and 1, which is sometimes modeled based on a linear slope or transition or line segment from 1 to 0, or vice versa.
In one embodiment, we use neural network on our Fuzzy system, in multiple different ways, e.g., using neural network to get the rules, or using neural network to do fuzzy logic inference, or using neural network to find and edit the membership functions or values for an object, or using neural network to construct a node combination structure based on fuzzy set connectives, e.g., union, intersection, and compensative connectives. In one embodiment, we use different aggregation operators to integrate membership values.
In one embodiment, we minimize the number of fuzzy rules, for efficiency, e.g., using rule pruning, rule combination, or rule elimination. In one embodiment, we eliminate the rules with low number of training samples or low reliability. In one embodiment, we use Karnaugh map to simplify the logic, including fuzzy rules. In one embodiment, we use data clustering to minimize fuzzy rules. In one embodiment, we use optimal defuzzification methods, e.g., using 2-layer neural network, or maximum matching, or centroid defuzzification, or maximum accumulated matching. These can be used e.g., for analyzing or recognition of maps, text, or handwriting.
In one embodiment, for learning machines, we use linear regression, least square, ridge regression, Widrow-Hoff algorithm, Support Vector Machines (SVM), Gaussian processes, Generalization technique (bounds on luckiness), or Kernel functions (to have a more general function for classification or cluster analysis), with SVM (and Kernel functions) at the center of our technique. Basically, in one embodiment, for the hyperplane separating the classes or clusters in the N-dimensional feature space, we want the largest distance from all neighboring points to the hyperplane, in average, as much as possible, as an optimization constraint. Or, in one embodiment, the separating hyperplane is defined as the plane that fits in between the growing neighboring points, as the growing neighboring points (from different clusters or classes) grow gradually in size or radius, as a point or sphere in the N-dimensional feature space, until there is no more room for their growth (and the growth stops at that point), with the separating hyperplane fitted in between the already-grown neighboring points (e.g., from the opposite clusters on two sides of the separating hyperplane)
In one embodiment, we use Vapnik's support vector machines (SVM) to classify the data or recognize the object. In one embodiment, in addition, we use kernels (e.g., using Gaussian processes or models) to be able to handle any shape of data distribution with respect to feature space, to transfer the space in such a way that the separation of classes or clusters becomes easier. In one embodiment, we use sparse kernel machines, maximum margin classifiers, multiclass SVMs, logistic regression method, multivariate linear regression, or relevance vector machines (RVM) (which is a variation of SVM with less limitations), for classification or recognition.
In one embodiment, for machine learning, the system starts with experiment generator describing the problem in hand. Then, the performance system (based on the history) feeds an analyzer, which provides the training examples to a generalizer module, which produces hypothesis for experiment generator, to complete the loop. In one embodiment, for machine learning, we use unsupervised learning or supervised learning or in combination, for different aspects of components of some data, es in an image, with many objects in it, for each object recognition, using different technique.
In one embodiment, for designing the learning program, the system first determines the type of training experience, followed by identification of target function, followed by determination of representation of learned function, and followed by learning algorithm, to complete the design.
In one embodiment, based on “Occam's razor” statement, we prefer the simplest hypothesis that fits the data. For example, a 10 node decision tree that fits the data completely is preferred over a 1000 node decision tree that fits the data completely, as well (due to the fact that it is less statistical coincidence, and more chance to fit or generalize correctly to future data).
In one embodiment, for machine learning, we use neural networks, perceptions, including gradient descent and delta rule, back propagation algorithm (including convergence and local minima problem), feedforward networks, hypothesis space search and inductive bias, with “Generalization” and “Overfitting” considerations, Q learning algorithm, or reinforcement learning, which all can be combined with our methods in this disclosure, as a complementary method, for improving the performance or efficiency.
In one embodiment, for classification, we may not be looking at enough dimensions for our feature space. So, randomly or based on historical data, the system guesses at the possible one or more extra dimensions to be added as new dimension(s) of feature space, and then tries to classify based on the new dimensions. If the result is better, based on separability of the clusters (or from their overlaps (which is generally an indication of not a good separation)), then we continue on that basis. Otherwise, we drop the new dimension from our space, and optionally, try another new dimension possibility, later on.
In one embodiment, for learning application, we use social collaborative filtering or adaptive lenses, to benefit from other people's experience, and to ada