complexobject_detection
Complex (or compound) objects are a collection of Simple Objects with a given relationship between them. At the moment, relationships are comparative (more, less) or spatial (being inside, being at a distance). With this tool, you can greatly expand the set of objects that can be recognized by the system. Often in the scientific literature for such a definition of objects, the term "scene" or "situation" is used, sometimes supplementing it with various constructions, for example, context. The image below is an example of a well-known picture (in Russia) that shows how relationships can be defined between objects.
The task of the algorithms for recognizing Complex Objects in this example can be "find a person with a rope" or "find a person standing on the road in front of a bush", "find two people holding hands." In the first case, the answer will be one group of objects, in the second two, in the third also two.
Similar to the problem of recognizing Simple Objects, recognition of Complex Objects should be performed in two modes: hard and soft.
- Hard - when the presence of all objects and connections is required for recognition.
- Soft - when groups of objects are taken that partially fall under the given description, and the confidence coefficient for this group is calculated.
At the moment, recognition is implemented only in hard mode.
Complex Object descriptions are located in the ComplexObjectBase tag, referring to Simple Objects from SimpleObjectBase and relationships from RelationLib.
As mentioned, relationships are specified in the RelationLib tag. For each relationship, a RelationShip tag is created with the following parameters:
- Name (string, must be set) The unique name of the relationship.
- Type (string, must be set) The relationship type must be one of the table below.
- Other parameters are specific to each type of relationship.
| Type | Description | Workspace* | Bilinear | Link |
|---|---|---|---|---|
| ImageRange | Distance ratio between objects in an image | I | ✔✔❌ (depends on the mode) | link |
| 3DRange | Distance relationship between objects in 3D space | 3D | ✔ | link |
| LogicAnd | Logical AND over two relationships | Inherited** | Inherited** | link |
| LogicOr | Logical OR over two relationships | Inherited** | Inherited** | link |
| LogicNot | Logical NOT to relation | Inherited** | Inherited** | link |
| SpaceIn | Checks that one object is completely inside another | I | ❌ | link |
| SpaceOut | Checks that one object is completely outside the other | I | ✔ | link |
| SpaceUp | Checks that the center of an object is higher than the center of another | I | ❌ | link |
| SpaceDown | Checks that the center of an object is lower than the center of another | I | ❌ | link |
| SpaceLeft | Checks that the center of an object is to the left of the center of another | I | ❌ | link |
| SpaceRight | Checks that the center of an object is to the right of the center of another | I | ❌ | link |
* - It can be either I, which means that the objects are compared in image coordinates, and 3D, which means that the relation operates in three-dimensional space, however, for this option, the objects must have attributes that determine the three-dimensional position.
** - Logical relationships take the parameters of the relationships to which they are applied.
All complex objects are described inside the ComplexObjectBase tag, and each of them must be enclosed in the ComplexObject tag.
- ID (int, must be set) A unique identifier for a complex object. May be the same as Simple Object identifiers.
- Name (string, must be set) The unique name of the Complex Object.
- SimpleObject - these tags denote which simple objects make up a complex
- Class (string) The name of a simple object in the base.
- InnerName (string) Some internal name that will be used when defining relations.
- Relation - these tags describe how simple objects are connected by relationships
- Obj1 (string) The internal name of the first object connected by the relationship.
- Obj2 (string) The internal name of the second object connected by the relationship.
- Relationship (string) Relation name from RelationLib.
An example of a complex object consisting of simple ones is a person with a recognized face. For this, the faces and figures of people are separately recognized, which are compared to each other according to the principle of finding a face in the rectangle of the entire figure. Such a complex object can be described in terms of the system as follows.
<?xml version="1.0" ?>
<AttributeLib>
<Attribute Name="HaarFace" Type="HaarCascade" Cascade="cascades/mallick_haarcascade_frontalface_default.xml"/>
<Attribute Name="CnnPerson" Type="Dnn" framework="tensorflow" weights="ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb" config="ssd_mobilenet_v1_coco_2017_11_17/config.pbtxt" labels="ssd_mobilenet_v1_coco_2017_11_17/mscoco_label_map.pbtxt" inputWidth="300" inputHeight="300" Probability="0.5" obj_id="1"/>
</AttributeLib>
<SimpleObjectBase>
<SimpleObject Name="Face" ID="10">
<Attribute Type="Detect">HaarFace</Attribute>
</SimpleObject>
<SimpleObject Name="CnnPerson" ID="67">
<Attribute Type="Detect">CnnPerson</Attribute>
</SimpleObject>
</SimpleObjectBase>
<RelationLib>
<RelationShip Type="SpaceIn" Name="in"/>
</RelationLib>
<ComplexObjectBase>
<ComplexObject ID="10" Name="FacedPerson">
<SimpleObject Class="CnnPerson" InnerName="Person"/>
<SimpleObject Class="Face" InnerName="Face"/>
<Relation Obj1="Face" Obj2="Person" Relationship="in"/>
</ComplexObject>
</ComplexObjectBase>
