Query Aware Determinization of Uncertain Objects

IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 27, NO. 1, JANUARY 2015 207
Query Aware Determinization of Uncertain
Objects
Jie Xu, Dmitri V. Kalashnikov, and Sharad Mehrotra, Member, IEEE
Abstract—This paper considers the problem of determinizing probabilistic data to enable such data to be stored in legacy systems
that accept only deterministic input. Probabilistic data may be generated by automated data analysis/enrichment techniques such as
entity resolution, information extraction, and speech processing. The legacy system may correspond to pre-existing web applications
such as Flickr, Picasa, etc. The goal is to generate a deterministic representation of probabilistic data that optimizes the quality of the
end-application built on deterministic data. We explore such a determinization problem in the context of two different data processing
tasks—triggers and selection queries. We show that approaches such as thresholding or top-1 selection traditionally used for
determinization lead to suboptimal performance for such applications. Instead, we develop a query-aware strategy and show its
advantages over existing solutions through a comprehensive empirical evaluation over real and synthetic datasets.
Index Terms—Determinzation, uncertain data, data quality, query workload, branch and bound algorithm
1 INTRODUCTION
WITH the advent of cloud computing and the
proliferation of web-based applications, users often
store their data in various existing web applications. Often,
user data is generated automatically through a variety
of signal processing, data analysis/enrichment techniques
before being stored in the web applications. For exam-
ple, modern cameras support vision analysis to generate
tags such as indoors/outdoors, scenery, landscape/portrait,
etc. Modern photo cameras often have microphones for
users to speak out a descriptive sentence which is then
processed by a speech recognizer to generate a set of
tags to be associated with the photo [1]. The photo
(along with the set of tags) can be streamed in real-time
using wireless connectivity to Web applications such as
Flickr.
Pushing such data into web applications introduces a
challenge since such automatically generated content is
often ambiguous and may result in objects with probabilis-
tic attributes. For instance, vision analysis may result in tags
with probabilities [2], [3], and, likewise, automatic speech
recognizer (ASR) may produce an N-best list or a confusion
network of utterances [1], [4]. Such probabilistic data must
be “determinized" before being stored in legacy web appli-
cations. We refer to the problem of mapping probabilistic
data into the corresponding deterministic representation as
the determinization problem.
• The authors are with the Department of Computer Science, Bren School
of Information and Computer Science, University of California at Irvine,
Irvine, CA 92617 USA. E-mail: {jiex, dvk, sharad}@ics.uci.edu.
Manuscript received 31 Jan. 2013; revised 18 Sep. 2013; accepted
18 Oct. 2013. Date of publication 3 Nov. 2013; date of current version
1 Dec. 2014.
Recommended for acceptance by L. Chen.
For information on obtaining reprints of this article, please send e-mail to:
reprints@ieee.org, and reference the Digital Object Identifier below.
Digital Object Identifier 10.1109/TKDE.2013.170
Many approaches to the determinization problem can
be designed. Two basic strategies are the Top-1 and
All techniques, wherein we choose the most probable
value / all the possible values of the attribute with
non-zero probability, respectively. For instance, a speech
recognition system that generates a single answer/tag for
each utterance can be viewed as using a top-1 strat-
egy. Another strategy might be to choose a threshold τ
and include all the attribute values with a probability
higher than τ. However, such approaches being agnos-
tic to the end-application often lead to suboptimal results
as we will see later. A better approach is to design
customized determinization strategies that select a deter-
minized representation which optimizes the quality of the
end-application.
Consider, for instance, an end application that supports
triggers/alerts on automatically generated content.
Examples of such an end-application includes pub-
lish/subscibe systems such as Google Alert, wherein users
specify their subscriptions in the form of keywords (e.g.,
“California earthquake") and predicates over metadata
(e.g., data type is video). Google Alert forwards all match-
ing data items to user based on the subscriptions. Now
consider a video about California Earthquake that is to be
published on Youtube. The video contains a set of tags that
were extracted using either automated vision processing
and/or information extraction techniques applied over
transcribed speech. Such automated tools may produce
tags with probabilities (e.g, “California": 0.9, “earthquake":
0.6, “election": 0.7), while the true tags of the video could
be “California" and “earthquake". The determinization
process should associate the video with appropriate tags
such that subscribers who are really interested in the video
(i.e., whose subscription includes the words “California
Earthquake") are notified while other are not overwhelmed
by irrelevant data. Thus, in the example above, the
determinization process should minimize metrics such
1041-4347 c 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

208 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 27, NO. 1, JANUARY 2015
TABLE 1
Notations
as false positives and false negatives that result from a
determinized representation of data.
Now consider a different application such as Flickr, to
which photos are uploaded automatically from cameras
along with tags that may be generated based on speech
annotation or image analysis. Flickr supports effective
retrieval based on photo tags. In such an application, users
may be interested in choosing determinized representation
that optimizes set-based quality metrics such as F-measure
instead of minimizing false positives/negatives.
In this paper, we study the problem of deteminizing
datasets with probabilistic attributes (possibly generated
by automated data analyses/enrichment). Our approach
exploits a workload of triggers/queries to choose the “best"
deterministic representation for two types of applications
– one, that supports triggers on generated content and
another that supports effective retrieval.
Interestingly, the problem of determinization has not
been explored extensively in the past. The most related
research efforts are [5], [6], which explore how to give
deterministic answers to a query (e.g. conjunctive selection
query [5]) over probabilisitc database. Unlike the prob-
lem of determinizing an answer to a query, our goal is
to determinize the data to enable it to be stored in legacy
deterministic databases such that the determinized repre-
sentation optimizes the expected performance of queries in
the future. Solutions in [5], [6] can not be straightforwardly
applied to such a determinization problem.1
Overall, the main contributions of this paper are:
• We introduce the problem of determinizing probabilis-
tic data. Given a workload of triggers/queries, the
main challenge is to find the deterministic represen-
tation of the data which would optimize certain qual-
ity metrics of the answer to these triggers/queries
(Section 2).
• We propose a framework that solves the problem
of determinization by minimizing the expected cost
of the answer to queries. We develop a branch-
and-bound algorithm that finds an approximate
1. We will show that in terms of quality of query answer, our pro-
posed determinization techniques are very close to those applied only
over probabilistic databases, which are not supported by many legacy
applications.
Fig. 1. Running example.
near-optimal solution to the resulting NP-hard prob-
lem (Section 3).
• We address the problem of determinizing a collection
of objects to optimize set-based quality metrics, such
as F-measure. We develop an efficient algorithm that
reaches near-optimal quality (Section 4).
• We extend the solutions to handle a data model
where mutual exclusion exists among tags. We show
that correlations among tags can be leveraged in our
solutions to get better results. We also demonstrate
that our solutions are designed to handle various
types of queries. (Section 5).
• We empirically demonstrate that the proposed algo-
rithms are very efficient and reach high-quality results
that are very close to those of the optimal solution.
We also demonstrate that they are robust to small
changes in the original query workload (Section 6).
2 PRELIMINARY
Before we formally define the determinization problem, let
us first introduce some important notation. The notation is
summarized in Table 1. We will also illustrate it using the
running example shown in Fig. 1.
2.1 Notations
Object. An uncertain object O is annotated with proba-
bilistic attributes. It could be, for instance, an output of
an entity resolution algorithm, or an image annotated by
probabilistic tags generated by a speech recognizer from
a provided speech annotation. Formally, an object O has
associated with it the following information:
1) W = {w1, w2, . . . , w|W|}, the set of uncertain tags
provided by some data processing technique, e.g.,
speech recognition, or entity resolution.
2) P = {p1, p2, . . . , p|W|}, where each pi ∈ P is the prob-
ability that the corresponding uncertain tag wi ∈ W
belongs to the ground truth tags G that will be
explained shortly. These probabilities are provided
by the data processing techniques.
We call this object model the tag model. Fig. 1 illustrates
our running example of an uncertain object O repre-
sented in the tag model. The set of uncertain tags of O is
W = {pier, beach, dog}. The corresponding probability set
is P = {0.35, 0.60, 0.70}. Initially, we make an assumption
that tags are independent. Later we present an approach
for handling correlation among tags and a more general
XOR model where mutual exclusions exist among tags.
Ground truth tags. The set of ground truth tags G
include the real tags that should have been associated with

XU ET AL.: QUERY AWARE DETERMINIZATION OF UNCERTAIN OBJECTS 209
the object, if the data processing technique outputted 100%
accurate results. For instance, in the case where objects
are speech annotated images, the ground truth tags cor-
respond to the actual spoken words. The ground truth tags
are unknown to the algorithm and are hence not part of
the uncertain object. These tags are only used for describ-
ing the quality metrics we will define later. For the object
in the running example, the ground truth tags could be
G = {dog, beach}. Notice that due to the uncertainty in the
results returned by the data processing techniques being
used, the set of uncertain tags W could be different from
the set of ground truth tags G.
Deterministic representation. A determinization pro-
cess for an object O is to select a deterministic representa-
tion to represent O, such that it can be stored in a legacy
system that does not support probabilistic input. We define
a deterministic representation (or answer set) of an object O
as a set of deterministic tags A = {w1, w2, · · · , w|A|}. For
example, one possible deterministic representation for the
object in the running example is A = {pier, beach}.
When it is not clear from the context, we will use sub-
script ‘O’ in variables W, G and A to indicate which object
they refer to, e.g. WO refers to “W of object O".
Query workload. A query workload is a set of queries
Q = {Q1, Q2, · · · , Q|Q|} that will be executed over the deter-
ministic representation. For clarity of presentation, we will
initially assume that each query Q ∈ Q is a conjunctive
query. We will later discuss how to extend it to other types
of queries. A conjunctive query Q ∈ Q has the following
information associated with it:
1) T = {q1, q2, . . . , q|T|} stores the set of terms associ-
ated with Q = q1∧q2∧· · ·∧q|T|. Given a deterministic
representation AO of an object O, the object satisfies
a conjunctive query Q iff T ⊆ AO.
2) f ∈ [0, 1] encodes the query weight that cor-
responds to the relative frequency of Q in the
workload.
We assume that, in general, such a workload is generated
from a past query log, modeled based on the past querying
patterns [7], or derived from frequent terms in past corpus
of data [8]. We will use subscript ‘Q’ for variables T and f to
indicate which query they refer to, e.g. fQ refers to “weight
of query Q". The running example in such that it can be
stored in a legacy system that does not Fig. 1 demonstrates
a workload of 6 conjunctive queries. The query terms and
their weights are also shown.
2.2 Quality Metrics
Given deterministic representation for each uncertain object
O ∈ O, we can define quality metric F(O, Q) of the
deterministic dataset with regard to a given query work-
load Q. The choice of the metric depends upon the end
application.
Cost-based metric. For applications that support trig-
gers/alerts, quality can be measured in terms of costs
of false negatives and false positives. We will denote by
GQ the set of objects that satisfy query Q, when the
deterministic representation of each object in O consists
of true/correct tags. We will denote by AQ the set of
objects that satisfy query Q, based on the deterministic
representations selected by an algorithm for objects in O.
Observe that for an object O, the case where O ∈ AQ but
O ∈ GQ corresponds to a false positive caused by the algo-
rithm. In this case, the query will retrieve an irrelevant
object. On the other hand, the case where O ∈ AQ but
O ∈ GQ corresponds to a false negative. In this case, the
query will miss a relevant object.
In the application where the cost-based metric is used,
each query is associated with two additional pieces of
information: (1) c+, the non-negative cost (penalty) of a
false positive; (2) c−, the non-negative cost (penalty) of a
false negative. We will assume that the default costs of
a false positive and a false negative are the same, e.g.,
c+ = c− = 1. For generality, however, we will keep c+ and
c− in the formulas in case they need to be different.2 Again,
we will use subscript ‘Q’ in c+, and c− to indicate which
query they refer to.
If tags A are chosen as the answer set of object O,
whereas the ground truth tags are G, then the cost of query
Q with respect to object O is defined as:
cost(O, Q) = cost(AO, GO, Q), (1)
where
cost(A, G, Q) =
⎧
⎨
⎩
c+
Q if O ∈ AQ ∧ O ∈ GQ,
c−
Q if O ∈ AQ ∧ O ∈ GQ,
0 otherwise.
(2)
The cost associated with the entire query workload Q with
respect to object O is defined as:
cost(O, Q) =
Q∈Q
fQ · cost(O, Q). (3)
Then the cost-based metric is defined as:
cost(O, Q) =
O∈O
cost(O, Q). (4)
For instance, assume that in Fig. 1 the ground truth tags
for the uncertain object are G = {dog, beach}. Suppose that
the algorithm chooses A = {beach, pier} as its answer set for
O. Then, queries Q3 and Q4 in Fig. 1 will cause two false
positives, on the other hand, Q1 and Q6 will result in two
false negatives. Thus, the cost with respect to O as defined
by Eq. (3) is 0.75.
Set-based quality metric. Applications that support
effective query-driven retrieval of objects often utilize set-
based quality metrics, such as F-measure or Jaccard simi-
larity. While we will focus on F-measure, our approaches
can be naturally applied to many other set-based metrics.
If AQ is the set of objects that satisfy query Q, based on
the algorithm, the F-measure of answers to query Q, with
respect to the set of objects O is defined as:
Fα(O, Q) = Fα(AQ, GQ), (5)
2. For example, three-level low/medium(default)/high model can
be used for costs. If a user of, say, a pub/sub system does not want to
miss notifications about items that include certain tags, (s)he can set
c− to “high" value. On the other hand, if (s)he does not want to be
flooded with objects that are not relevant to his/her interest, (s)he can
set c+ to be “high" for the corresponding query.

where
Fα(A, G) =
(1 + α) · Precision(A, G) · Recall(A, G)
α · Precision(A, G) + Recall(A, G)
(6)
where
Precision(A, G) =
|A G|
|A|
, (7)
Recall(A, G) =
|A G|
|G|
. (8)
The set-based metric is then defined as:
Fα(O, Q) =
Q∈Q
fQ · Fα (O, Q) . (9)
2.3 Determinization Problem
Having defined the notation, we now can define the deter-
minization problem:
Definition 1 (Determinization). Given a set of uncertain
objects O = {O1, O2, . . . , O|O|}, a query workload Q and
a quality metric F, the goal of the deteriminization prob-
lem is for each object O ∈ O to select from WO a set of tags
AO ⊆ WO as the deterministic representation of O, such that
F(O, Q) is optimized.
The ground truth tags G are not associated with uncer-
tain objects and thus not known to any approach to the
determinization problem. Therefore, such algorithms can-
not directly measure the quality F(O, Q) during their exe-
cution. Thus, in the following section, we will present our
solution that is based on maximizing the expected quality
measure E(F(O, Q)).
Before we describe our solution to the determinization
problem, we note that a baseline algorithm for deter-
minization is a thresholding (threshold-cut) approach. The
thresholding approach employs a pre-defined threshold τ.
For each object O it composes its answer set A by choosing
wi tags from W such that P(wi) ≥ τ. The advantage of this
approach is that it is computationally efficient and poten-
tially can be tuned to a particular dataset O and workload
Q by changing τ. The main drawback of this approach is
that it is unaware of the query workload (“query-unaware")
and thus does not necessarily optimize the given quality
metrics, which leads to lower quality.
3 DETERMINIZATION FOR THE COST-BASED
METRIC
When the cost-based metric is used, F(O, Q) is evaluated
using cost(O, Q) measure described in the previous sec-
tion. Thus, the goal of the determinization problem for
the cost-based metric is for each object O ∈ O to choose
tags AO ⊆ WO as its deterministic representation, such that
cost(O, Q) is minimized. It is easy to see that:
min
A1,A2,...,A|O|
cost(O, Q) =
O∈O
min
AO
cost(O, Q). (10)
The task thus reduces to determining for each object O ∈ O
its answer set AO independently from other objects, such that
cost(O, Q) specified by Eq. (3) is minimized.
TABLE 2
Expected Cost for All Possible Answer Sets
3.1 Expected Cost
The challenge of minimizing cost(O, Q) arises since the
ground truth G is not known and thus cost(O, Q) cannot
be computed directly. The idea of our solution is to choose
the answer set based on expected cost. Namely, let W be
the space of all possible ground truth sets for O, given the
uncertain tags in W. Let G ∈ W be an instance of a possible
ground truth. Table 2 lists all the 8 possible deterministic
representations of the uncertain object in our running exam-
ple. A possible ground truth instance G can be any one of
the 8 representations. Let P(G = GO) be the probability that
the instance G is the actual ground truth of object O. Then,
if we choose some deterministic representation AO for O,
we can compute the expected cost of a query Q with respect
to O as:
E(cost(O, Q)) =
G∈W
cost(AO, G, Q) · P(G = GO). (11)
Directly computing expected cost using Eq. (11) is expo-
nential in the number of tags in W and thus impractical.
Let us define PO∈GQ = P(O ∈ GQ) as the probability that
object O satisfies query Q, based on ground truth tags of
O.3 The following lemma provides an efficient way to com-
pute the expected cost without enumerating every possible
G ∈ W. The proof is given in Appendix A, which is avail-
able in the Computer Society Digital Library at http://
doi.ieeecomputersociety.org/10.1109/TKDE.2013.170.
Lemma 1. E(cost(O, Q)) for a particular chosen answer set A
can be computed as:
E(cost(O, Q)) = E(cost(A, G, Q)) (12)
where
E(cost(A, G, Q)) =
c+
Q · (1 − PO∈GQ ) if O ∈ AQ;
c−
Q · PO∈GQ if O ∈ AQ.
(13)
For query workload Q we have:
E(cost(O, Q)) = E(cost(A, G, Q)), (14)
where
E(cost(A, G, Q)) =
Q∈Q
fQ · E(cost(A, G, Q)). (15)
Given Eq. (14), the optimal answer set, in the expected
sense, can be determined as:
Aopt = argmin
A∈W
E(cost(O, Q)). (16)
3. For example, under the assumption that tags in object O
are independent and query Q is a conjunctive query, PO∈GQ
=
qi∈TQ
P(qi ∈ GO).

TABLE 3
Notation for Node v
While Aopt is not necessarily equal to the ground truth G,
it is the best answer set given that the ground truth can be
any instance G from W with the corresponding probability
of P(G = G). Thus our goal reduces to being able to effi-
ciently find Aopt. We refer to the problem of finding Aopt that
minimizes E(cost(O, Q)) as the Expected Determinization
problem for the Cost-based Metric (EDCM). We prove in
Appendix B, available online, that:
Lemma 2. EDCM is NP-hard.
EDCM remains NP-hard even under some simplified
special cases, such as (a) weights of all queries are the
same (it is already shown in the proof in Appendix B,
available online); (b) each query has the same (unit) cost
of false positives and false negatives. This case can also
be proven to be NP-hard using a similar idea to that of
Appendix B, available online, but by a reduction from the
weighted MAX-SAT problem.4
To solve EDCM, we next will explain a naive
enumeration-based algorithm which finds the exact solu-
tion to the problem. This naive algorithm is, however,
exponential in the number of tags associated with the
object. Therefore, it is infeasible when the number of tags
is large, which is the case in multiple data sets we will
use in Section 6. Hence, we develop a branch-and-bound
algorithm to solve EDCM approximately. In Section 6,
we empirically demonstrate that our approximate solution
reaches the quality that is very close to the exact one, while
being orders of magnitude more efficient.
3.2 Enumeration-Based Optimal Approach
The desired Aopt, as defined by Eq. (16), can be found by
enumerating all possible answer sets and picking the one
with the lowest expected cost. For each uncertain tag w ∈ W
we have two choices: either include it in the answer set or
not and thus there are 2|W| possible answer sets. Hence,
while this algorithm is expected to produce high quality
(low cost) results, it is exponential on |W|, and therefore it is
impractical from the computational efficiency perspective.
For instance, Table 2 shows the expected costs with
all possible answer sets for the running example. The
4. EDCM can be solved in polynomial time under very specific
conditions such as all queries contain single distinct terms but such
situations are unrealistic.
Fig. 2. Branch and bound algorithm.
enumeration-based approach will go over all the 8 possibili-
ties and select {dog}, since it has the minimum expected cost
of 0.311. For the thresholding approach, if we set τ = 0.55
as the threshold, it will output the answer set {dog, beach},
which ranks only the 5th among the eight alternatives.
3.3 Branch and Bound Algorithm
Instead of performing a brute-force enumeration, we can
employ a faster branch and bound (BB) technique. The
approach discovers answer sets in a greedy fashion so that
answer sets with lower cost tend to be discovered first.
Table 3 summarizes the notations we will use to illustrate
the proposed BB algorithm.
3.3.1 Outline of the BB algorithm
The outline of the proposed BB algorithm is illustrated in
Fig. 2. The algorithm uses a branch and bound strategy to
explore a search tree. Fig. 3 shows a complete search tree for
the running example. Each node v in the tree corresponds to
a partial tag selection where decisions of whether to include
certain tags in the answer set or not have been made. We
capture the partial selection of tags using the concept of
sequence defined as follow:
Definition 2 ((Sequence)). A sequence is defined as a map-
ping S:W → {yes, no, undecided}, which maps each tag
w ∈ W into three categories:
1) S[w] = yes, the algorithm decides to put w in A;
2) S[w] = no, the algorithm decides not to put w in A;
3) S[w] = undecided, the algorithm has not yet decided
whether to put w into A or not.
Sequence S is an answer sequence if it does not contain
undecided tags, and maps uniquely into one particular answer
set A = A(S), where for each w ∈ W it holds w ∈ A iff
S[w] = yes and w ∈ A iff S[w] = no.
We denote by Sv the sequence that corresponds to node
v. Note that the root of the tree corresponds to a sequence
where all tags are undecided and each leaf node corre-
sponds a possible answer sequence. The algorithm starts
5. Intuitively, τ = 0.5 is a very good choice of a threshold value,
as it keeps tags that have at least 50% chance of being in the ground
truth and removes the rest.

Fig. 3. Search tree for the running example. For each node v that
is visited, three pieces of information are listed. (1) Sh
v , upper bound
sequence. (2) lv and hv , lower bound and upper bound. (3) Qv ,
undecided queries.
with a priority queue H that consists of only the root node.
At any step, the algorithm extracts from the priority queue
the best node v, associated with a lower bound lv and an
upper bound hv of the lowest cost achievable from expand-
ing this node. If the lower bound lv is larger than the
lowest cost c∗ found so far, the node can be pruned (Step 7).
Otherwise, the algorithm selects an uncertain tag (Step 13),
creates two branches corresponding to the the decision to
map the tag to yes/no and computes lower and upper
bounds of the two nodes generated during the branching
procedure (Step 14). The execution time of the algorithm
is controlled and bounded by using the Nleaf parameter,
where the algorithm is not allowed to explore more than
Nleaf answer sets (Step 11).
The performance of the algorithm depends upon the
choice of the node (amongst all the non-leaf nodes) to
branch, the computation of upper and lower bounds of a
node, and the choice of the tag used to expand the node. We
next describe these components of the algorithm in detail.
3.3.2 Node Selection
The algorithm maintains a priority queue H for picking
a node v that contains the most promising sequence Sv
to continue with. Among the nodes in the priority queue,
which one should we choose to branch next? Let us con-
sider sequence Sv that corresponds to a node v ∈ H. For v
we define Av as the set of answer sequences that correspond
to the leaf nodes derivable from node v via the branching
procedure described above. That is, Av corresponds to the
leaf nodes of the subtree rooted at node v. For example,
for node v2 of the tree in Fig. 3, Av2 = {Sv8 , Sv9 , Sv10 , Sv11 }.
Then for node v let
mv = min
A∈Av
E(cost(A, G, Q)) (17)
be the value of the minimum expected cost among these
sequences. Notice that if we knew the exact value of mv,
it would have been an ideal key for the priority queue H,
since it would lead to the quickest way to find the best
sequences. The problem is that the exact value of mv is
unknown when v is branched, since the subtree rooted at
v is not yet constructed at that moment.
Even though mv is unknown it is possible to quickly
determine good lower and upper bounds on its value
v ≤ mv ≤ hv, without comparing scores of each sequence
in Av. For any node v, if Sv is an answer sequence then
the bounds are equal to the cost of the sequence itself,
otherwise the bounds are computed as explained next.
3.3.3 Upper Bound
To compute an upper bound on mv, it is sufficient to
pick one answer sequence A from Av and then set hv =
E(cost(A, G, Q)). We refer to such an answer sequence as
the upper bound sequence Sh
v for node v. The procedure for
choosing Sh
v determines the quality of the upper bound.
The proposed algorithm employs a greedy strategy to
compute upper bound sequence, as illustrated in Fig. 5.
It traverses all w ∈ Sh
v in a particular order that we will
explain later. For each undecided tag w, it assigns to
Sh
v[w] the best yes/no decision that results in the minimum
expected cost of query workload. We observe that the deci-
sion for one tag w will affect only the results of those queries
whose terms include w, which we refer to as Qw. Thus, the
greedy strategy makes best decision for each undecided
tag w ∈ Sh
v such that overall expected cost of queries in Qw
is minimized.
Notice that once the algorithm determines the values of
all w ∈ Sh
v, the procedure EXP-COST in Step 5 and 7 actually
computes expected cost using Eq. 13. However, the chal-
lenge of computing expected cost arises since Sh
v is not an
answer sequence until the end of the greedy algorithm.
Eq. 13 is not applicable in that case, because there might
exist some undecided tag in Sh
v and the algorithm can not
yet determine whether Q will be satisfied. To resolve this
challenge, we rewrite Eq. 13 as follows:
E(cost(S, G, Q)) = c+
Q · (1 − PO∈GQ ) · PO∈SQ (18)
+c−
Q · PO∈GQ · (1 − PO∈SQ ),
where PO∈SQ is the probability that object O satisfies query
Q based on the associated sequence S. Notice that if S is
an answer sequence, then PO∈SQ will be either 1 or 0 and
Eq. 18 is actually equivalent to Eq. 13. While S is not yet
an answer sequence, we make an assumption that for each
undecided tag w ∈ S, P(w ∈ A), which denotes the prob-
ability that w will be included in answer set A, is equal to
P(w ∈ G). Thus, according to the value of S[w], P(w ∈ A) is
computed in the following three cases:
1) iff S[w] = yes, then P(w ∈ A) = 1.
2) iff S[w] = no, then P(w ∈ A) = 0.
3) iff S[w] = undecided, then P(w ∈ A) = P(w ∈ G).
Thus for a conjunctive query Q, PO∈SQ can be computed as
PO∈SQ = qi∈TQ
P(qi ∈ A).
Observe that if v is a leaf node, its sequence Sv and upper
bound sequence Sh
v will be identical. Also notice that an
upper bound sequence Sh
v is an answer sequence even if the
node v is not a leaf node. Therefore every time upper bound
sequence of a node is changed during branching procedure,
we “observe" a new answer sequence. The algorithm stops
after observing Nleaf answer sequences and returns the best
answer set observed thus far.
3.3.4 Lower Bound
As defined by Eq. (15), the expected cost E(cost(A, G, Q))
of answer set A is computed over all the queries in query

Fig. 4. Branching procedure.
workload Q. One answer set A may be the best one for
certain queries in Q, but not for the others. To compute a
lower bound on mv, we find the answer set with lowest
expected cost for each query independently and use the sum
of these costs as the lower bound. That is, for a given node v
Q∈Q
fQ · min
A∈Av
E(cost(A, G, Q)) ≤ min
A∈Av
E(cost(A, G, Q)).
Thus, Q∈Q fQ · minA∈Av E(cost(A, G, Q)) can be used as
the lower bound on mv, and the task becomes to compute
minA∈Av E(cost(A, G, Q)) for each Q ∈ Q.
For a given node v and query Q, there are three
different cases for computing the lowest expected cost
minA∈Av E(cost(A, G, Q)), depending on Q and the sequence
Sv that corresponds to v:
1) Query Q will retrieve object O based on Sv.6 Thus
only a false positive is possible and the expected
cost can be already computed as c+
Q · (1 − PO∈GQ ),
even if Sv itself is not yet an answer sequence.
2) Query Q will not retrieve object O based on Sv.7
Thus only a false negative is possible. Like Case 1,
without knowing the decision for undecided tags in
Sv, we can already compute the expected cost for Q
as c−
Q · PO∈GQ .
3) It is undecided whether Q will retrieve object O or
not.8 Thus, the best answer set derived from S is
the one that leads to the minimum cost. Therefore,
the lowest expected cost is min(c+
Q · (1 − PO∈GQ ), c−
Q ·
PO∈GQ ).
The computed values of lower and upper bounds can
be used to estimate mv. Such estimation can serve as the
key in the priority queue H. The upper bound hv is the
expected cost of the answer set determined by the greedy
algorithm shown in Fig. 5. While not necessarily equal to
mv, it is very close to mv in our experiment. Therefore, we
employ the upper bound as the key. The lower bound and
upper bound are also utilized to prune the search space, as
explained before.
3.3.5 Choosing a Tag to Branch Next
Once a node v is chosen, we need to choose one undecided
tag of the node to branch next. The intuition is that if tag
w we choose is a common term for the queries in work-
load, then making a decision for w will help to quickly
determine the expected cost of many queries. Hence, the
6. For a conjunctive query, the test is ∀q ∈ TQ, Sv[q] = yes.
7. For a conjunctive query, the test is ∃q ∈ TQ s.t. Sv[q] = no.
8. For a conjunctive query, the test is ∃q ∈ TQ s.t. Sv[q] =
undecided and ∃q ∈ TQ s.t. Sv[q] = no.
Fig. 5. Procedure for computing upper bound sequence.
algorithm sorts all tags of Sv by frequency of occurrence.
Every time it needs to choose an undecided tag to branch
next, it chooses the one that appears in most of the queries
in the workload. As a result, the greedy algorithm in Fig. 5
also follows the same order for determining values of
undecided tags.
3.4 Query-Level Optimization
The performance of the BB algorithm can be significantly
improved further by employing query-level optimizations.
For a given sequence Sv of node v, we might be able to
exactly determine the expected cost of certain queries (e.g.,
Cases 1 and 2 above), even if Sv is not an answer sequence.
In other words, for these queries we can get their cost with-
out expanding Sv into the corresponding answer sequences
A ∈ Av. We refer to such cost as fixed cost of v, and to such
queries as fixed or decided queries with respect to O. Thus,
each node v is also associated with the set of undecided
queries Qv, its fixed cost ˆcv, as well as its lower and upper
bounds ˆlv and ˆhv on non-fixed cost. Note that lv = ˆcv + ˆlv
and hv = ˆcv + ˆhv. Fixed cost and fixed queries give us an
opportunity to make the BB algorithm even more efficient
by reducing the number of queries involved in computing
the expected cost.
When the algorithm performs a branching at a node,
as illustrated in Fig. 4, it creates two new nodes vyes and
vno, by making yes/no decision for one undecided tag w.
The procedure in Fig. 6 is then used to compute the lower
bound and upper bound of each new node. Since one addi-
tional tag is now decided for each new node v, it is possible
that expected cost for some query Q ∈ Qv can be computed
(Case 1 and 2). If so, Q becomes a fixed query for the new
node and will not be considered in future computations
derived from that node. Based on the relation between Sv
and Q, there are three different cases for updating ˆcv, ˆlv
and ˆhv:
1) Query Q will retrieve object O based on Sv. The
cost of Q can only be caused by a false positive
and Q becomes a fixed query. Thus, the lower bound
on non-fixed cost ˆlv and the fixed cost ˆcv will be
updated as follow:
ˆlv←ˆlv − fQ · min(c+
Q · (1 − PO∈GQ ), c−
Q · PO∈GQ ) (19)
ˆcv←ˆcv + fQ · c+
Q · (1 − PO∈GQ ) (20)

Fig. 6. Procedure for updating node information, including lower and
upper bounds.
Since one query has become fixed, the upper
bound on non-fixed cost ˆhv needs to be recomputed
(Step 14).
2) Query Q will not retrieve object O based on Sv. The
cost of Q can only be caused by a false negative
and Q becomes a fixed query. In this case, ˆlv will
be updated using Eq. 20 and ˆcv will be updated as
follow:
ˆcv ← ˆcv + fQ · c−
Q · PO∈GQ . (21)
Like in case 1, ˆhv needs to recomputed as well.
3) Q is still not decided for object O. In this case, neither
ˆcv nor ˆlv needs to be updated.
This optimization technique trades memory consump-
tion for computational efficiency, as each node v in priority
queue H maintains the set of undecided queries Qv. Thus,
when the amount of memory is limited, this optimization
technique should not be applied. However, it works well in
practice and does not require much memory space. This is
because typically few queries in Q are relevant to a partic-
ular objects O, e.g. only those that contain tags from W.
Furthermore, each yes/no decision tend to convert lots
of undecided queries into decided ones, thus |Qv| reduces
significantly as the algorithm moves toward the leaf
nodes.
Example 1. Figs. 3 and 7 illustrate the search tree and the exe-
cution of the proposed BB algorithm for the running example.
Fig. 7 lists the values of three important variables reached at
the beginning of each iteration: (1) H, the priority queue; (2)
A∗, the best answer set found so far; and (3) c∗, the lowest
cost found so far. The super script ‘b’ means that the node
is to be branched in this iteration. The superscript ‘p’ means
that the node is to be pruned according to its lower bound
and the lowest cost c∗ found so far. The superscript ‘a’ means
that the node corresponds to an answer sequence and thus will
not be branched anymore. Because of the priority queue, the
algorithm quickly finds the best answer set {dog} after 3 iter-
ations. Due to the pruning strategy, only 3 out of 15 nodes
have been explored and branched: v1, v2, and v4.
Fig. 7. Illustration of the BB algorithm shown in Fig. 2.
3.5 Performance of Branch and Bound
Let Qv be the set of queries in query workload and T be
the maximum number of query terms, W be the number
of uncertain tags in the object, Nleaf be the limit on the
number of answer sets observed. In Appendix C, available
online, we prove that the computational complexity of the
BB algorithm is O(Nleaf(|Qv| + log Nleaf)).
As we will see in Section 6, even small values of Nleaf,
lead to nearly optimal quality of the BB algorithm on
large datasets in practice. This results in orders of magni-
tude of improvement of BB over the optimal enumeration
algorithm in terms of the efficiency.
3.6 Robust Workload
To make the solution resilient to small changes in the work-
load at runtime, the approach adds the default workload Qdef
to the original workload Qorig given to the algorithm, that is,
Q = Qorig ∪Qdef. It does so by using a Lidstone’s estimation-
like technique that creates a uniform prior on unobserved
queries by adding small values λ to query frequencies [9].
Let V be the vocabulary that contains all the distinct tags
found in the provided corpus of objects O. Then Qdef is con-
structed by creating a query Q = w for each word w ∈ V
with c+ = c− = 0.5 and f = λ for some small value of λ.
This ensures that all the words found in the object corpus
O are also covered in the query workload Q, while at the
same time the choice of tags for each objects O ∈ O will
still be largely determined by the original workload Qorig
since λ is small. We will see in Section 6 that this technique
makes the overall approach very robust to small changes
in the original query workload.
4 DETERMINIZATION FOR SET-BASED METRIC
Previous sections described a solution to the determiniza-
tion problem for the cost-based metric. In this section we
describe how to solve the problem for the set-based metrics.
4.1 Expected F-measure
Since the ground truth tags are not known, Fα(O, Q) in
Eq. 9 can not be computed directly. The idea is thus to
determinize objects based on expected F-measure. That is,
let WO be the space of all possible subsets of O. Each
G ∈ WO could be a possible set of objects that satisfy a
query Q based on ground truth tags. Thus, the expected
F-measure of a determinization result, with respect to query
Q, is computed as:
E(Fα(O, Q)) = E(Fα(AQ, GQ)), (22)
where E(Fα(AQ, GQ)) =
G∈WO
Fα(AQ, G)P(G = GQ). For a
query workload Q, we have

E(Fα(O, Q)) =
Q∈Q
fQ · E(Fα(O, Q)). (23)
Thus, our goal reduces to determinizing all objects O ∈ O,
such that E(Fα(O, Q)) is maximized. It is easy to prove that
this problem is also NP-hard.
4.2 Efficient approach for computing Expected
F-measure
For a particular query Q, each object O can either satisfy it
or not, so there are 2|O| possible ground truth sets for Q.
Simply enumerating over WO to compute E(Fα(AQ, GQ))
is infeasible. One observation is that while computing
expected F-measure with respect to a particular query Q,
we need to consider only those objects that can possibly
satisfy Q. We thus define candidate set as OQ = {O ∈
O:PO∈GQ > 0}. In practice, |OQ| is smaller than |O| and we
can reduce the size of enumeration space from 2|O| to 2|OQ|.
However, the computation is still exponential. We take
the approach proposed in [5], which uses Taylor series
expansion to approximate E(Fα(AQ, GQ)). The approxima-
tion of F-measure for a particular AQ is computed as:
Ê(Fα(AQ, GQ)) =
(1 + α) O∈AQ
PO∈GQ
|AQ| + α O∈OQ
PO∈GQ
. (24)
In [5], it is shown that Ê(Fα(AQ, GQ)) leads to highly
accurate results, especially for large values of |OQ|. Thus
we use a hybrid approach. When |OQ| is smaller than
a threshold τenum, it employs enumeration to compute
exact E(Fα(AQ, GQ)), otherwise, it takes the approximation
Ê(Fα(AQ, GQ)). We set τenum to be 6, as suggested by [5].
4.3 Iterative Algorithm
In this section, we propose an efficient iterative approach to
the determinization problem for the set-based metric. The
basic framework is outlined in Fig. 8. It first determinize
all objects (Step 3), using a query unaware algorithm, such
as threshold-based or random algorithm, followed by an
iterative procedure. In each iteration, the algorithm picks
one object Oi (Step 5). It then treats other objects O {Oi} as
already determinized, and determinizes Oi again such that
the overall expected F-measure E(Fα(O, Q)) is maximized
(Step 6-9). In this way, E(Fα(O, Q)) will either increase or
remain the same in each iteration. For every |O| iterations,
the algorithm checks the value of E(Fα(O, Q)), and stops
if the increase of the value since last check-point is less
than certain threshold. The main question is how to, in each
iteration, determinize the chosen object Oi such that the
overall expected F-measure is maximized.
4.3.1 Negative and Positive F-measure
To answer the question, let us first define two values for
each query Q: negative F-measure, referred as F−
Q, and pos-
itive F-measure referred as F+
Q. For a chosen object Oi, F−
Q
is defined as the expected F-measure with respect to Q if
Oi is excluded from AQ, that is F−
Q: = E(Fα(AQ {Oi}, GQ)).
Likewise, F+
Q is defined as the expected F-measure if Oi is
included in AQ, that is F+
Q: = E(Fα(AQ {Oi}, GQ)).
Example 2. Assume the candidate set for a query
Q is OQ = {O1, O2, O3, O4, O5, O6, O7} and the
Fig. 8. Efficient algorithm for set-based determinization.
corresponding probabilities of these objects satisfying Q
are {0.1, 0.5, 0.8, 0.3, 0.1, 0.1, 0.1}. Assume that in some
iteration, according to the determinization results of objects
in OQ, the set of objects satisfying Q is AQ = {O1, O2, O3}.
We can see that O∈AQ
PO∈GQ = 1.4, O∈OQ
PO∈GQ = 2
and |AQ| = 3. According to Eq. 24, the approximation of
F1-measure (F-measure for α = 1) is 2 × 1.4/(3 + 2) = 0.56.
Now assume the algorithm needs to compute F−
Q and
F+
Q for O1. Observe that AQ {O1} = {O2, O3}, thus
|AQ {O1}| = 2, O∈AQ{O1} PO∈GQ = 1.3 and the value of
F−
Q is 2 × 1.3/(2 + 2) = 0.65. Since AQ {O1} = AQ, the
value of F+
Q is already computed as 0.56.
4.3.2 Determinizing Individual Object
Having updated negative and positive F-measures for all
queries, we are left with the problem of how to deter-
minize the chosen object Oi such that the overall expected
F-measure of the query workload is maximized. This prob-
lem is virtually the same as the EDCM problem, where
the goal is to determinize an object such that the overall
expected cost of a query workload is minimized. Thus, we
can employ the Branch and Bound algorithm in Section 3.3.
More specifically, the BB algorithm can be applied with two
small modifications:
1) If we look into Eq. 13, c+
Q·(1−PO∈GQ ) is the expected
cost of query Q if object O is included in AQ and
c−
Q ·PO∈GQ is the expected cost if O is excluded from
AQ. Likewise, F+
Q is the expected F-measure with
respect to Q if object O is included in AQ and F−
Q is
the expected F-measure if O is excluded from AQ.
Therefore, while applying the BB algorithm, we just
need to replace c+
Q · (1 − PO∈GQ ) and c−
Q · PO∈GQ by
F+
Q and F−
Q respectively.
2) Since the original BB algorithm is to find the min-
imum, while our task here is to find the max-
imum, the BB algorithm needs to be changed
in a symmetric fashion (for example, exchanging
the ways to compute lower bound and upper
bound). The main structure of the algorithm stays
unchanged.
We call the modified BB algorithm as BBF. As shown in
Step 9 of Fig. 8, the BBF procedure is used to determinize
the chosen object Oi to maximize the overall expected

F-measure of query workload Q and return the resulting
maximum expected F-measure.
4.3.3 Picking Next Object
Another question is how to pick next object to determinize.
One strategy is for each object Oi ∈ O to look ahead
the overall expected F-measure resulted from choosing this
object. The object that leads to the maximum value is
chosen as the object to determinize. This strategy, though
ensuring maximum increase of the overall expected F-
measure in each iteration, will add a linear factor to the
overall complexity of the algorithm. Thus, it is not suitable
for large datasets. Another strategy is to simply loop over
the dataset or choose objects in a random order. Although
this strategy is not necessarily the best one in terms of
leading the algorithm towards convergence, it is a constant
operation. We thus employ the second strategy.
4.3.4 Performance of Iterative Algorithm
The computational complexity of the iterative algorithm
(Fig. 8) is determined by the while-loop. Each iteration of
the while-loop consists of computing F−
Q and F+
Q for each
query Q (Steps 6-8) and calling the BBF procedure (Step 9).
We can prove the following lemma (Appendix D, available
online):
Lemma 3. F−
Q and F+
Q for one query Q can be updated in
constant time.
Therefore, the complexity of each iteration is O(|Q| +
BBtime), where BBtime is the complexity of the branch and
bound algorithm described in Section 3.3. The overall com-
plexity of the iterative algorithm is then O(n(|Q| + BBtime)),
where n is the number of iterations before the algorithm
stops.
4.4 Other Set-Based Metrics
While we illustrate the algorithm using F-measure, the iter-
ative framework can also be used for optimizing other
set-based metrics, such as Jaccard distance and Symmetric
distance [6]. We can observe from Fig. 8 that instead of com-
puting F−
Q and F+
Q (Step 7, 8), the task is now to update
expected Jaccard distance or Symmetric distance in the two
cases where the chosen object Oi is included in AQ and not.
The remaining part of the algorithm can stay the same.
5 EXTENSIONS TO SOLUTIONS
In developing the solution, we have made some assump-
tions both about the data and the query, many of which
can be relaxed by appropriate extensions to the pro-
posed scheme. We show how to handle correlation among
tags and test the impact of leveraging the correlation in
Appendix E, available online. We show how to process
non-conjuctive queries specified in the form of DNF/CNF
in Appendix F, available online. In this section, we focus
on how to extend the proposed solution to handle mutual
exclusion amongst tags.
Techniques like entity resolution and speech tagging
may return probabilistic attributes that have mutually
exclusive values/tags. Specifically, one such typical repre-
sentation is when, one object O is annotated with a set of
Fig. 9. Example of XOR model.
probabilistic attributes {a1, a2, . . . , aK}. Each attribute ai is
given as a random variable that takes mutually exclusive
values ui ∈ Uai
with probability pi ∈ Pai
, where:
1) Uai
= {u1, u2, . . . , u|Uai
|} is an option tag set that
consists of mutually exclusive possible tags for ai.
2) Pai
= {p1, p2, . . . , p|Uai
|} stores the corresponding
probabilities, where pj is the probability that the
ground truth value of attribute ai is uj and j pj = 1.
We call this representation as mutual-exclusion (XOR) model.
Fig. 9 shows an example of XOR model. The object has 3
attributes {a1, a2, a3}. For instance, attribute a3 can take two
values, either sky with probability 0.8, or style with prob-
ability 0.2. Since the values are mutually exclusive, query
Q = sky ∧ style should not retrieve this object.
Observe that the tag model from Section 2 is actually a
special case of the XOR model. Namely, each pair (wi, pi)
in the tag model corresponds to one attribute ai in the XOR
model associated with an option set Ui = {wi, null} and a
probability set Pai
= {pi, 1−pi}, where null means no option
tag of the attribute is in the ground truth.
Based on above observation, both BB and iterative algo-
rithms can be naturally extended to handle objects in XOR
model as follows:
1) Sequence. Instead of mapping each attribute to
yes, no, or undecided, a sequence now maps each
attribute a into either undecided or one option tag
u ∈ Ua.
2) Branching. While branching on a tree node v, instead
of making yes/no decisions, the algorithm will
now create one new node vu for each u ∈ Ua, where
vu is equal to v except that the sequence of vu now
maps attribute a into u. As a result, the search tree
is no longer binary in general.
3) Computing PO∈GQ . The way to compute PO∈GQ will
need to be adjusted accordingly. In general, there
is known result [5] that if the number of the query
terms n in a conjunctive query Q = q1, q2, . . . , qn
is bounded by a constant, then for XOR model
PO∈GQ can be computed in O(K) time where K is
the number of probabilistic attributes of object O.
6 EXPERIMENTAL EVALUATION
In this section we empirically evaluate the proposed
approaches in terms of both the quality and efficiency.
6.1 Experimental Setup
Datasets.
1) RealSpeech consists of 525 speech segments, each
containing a sentence from broadcast news such as

VOA. This dataset imitates speech annotation of
uploaded videos. Ground truth tags for this dataset
are given, thus making it possible to test the qual-
ity of various approaches. The average number of
words in one sentence is 20. We applied an ASR
ensemble approach to combine two speech recog-
nizers to get a tag model representation for each
speech segment.
2) PubData consists of 11.7K publications and
14.6K authors derived from CiteSeer and HPSearch
datasets. The objects in it are publications and
attributes are references to authors in these publi-
cations, which can be ambiguous. The references
are resolved using the entity resolution method
from [10]. It generates the list of alternatives for
each attribute as well as their probabilities. Notice
that objects in this dataset are represented in XOR
model.
3) LargeImg This semi-real dataset consists of 60, 000
real tagged Flickr images. Each image is tagged with
8 words on average. The probabilistic attributes for
this dataset have been generated synthetically from
these real tags as follows: For each ground truth tag
g ∈ G, we generate an attribute with N option tags.
The probabilities p1, p2, . . . , pN for N option tags are
assigned by generating N random numbers drawn
from uniform distribution in (0, 1] and normaliz-
ing them such that they add up to 1. The interval
method in [5] is then used to decide which option
tag ui is the ground truth value for the attribute,
such that the chance of ui being the ground truth
is pi.
Query Workload. We test our solution over several syn-
thetic query workloads where parameters of workloads
follow certain distributions, similar to [11], [12]. While
we experiment with several query workloads (discussed
later), most of the experiments were conducted on the main
query workload Qorig that consists of single- and multi-term
queries. Similar to [5], to make queries more meaningful,
we pick combinations of single and multiple terms whose
co-occurrence count in the whole corpus of objects exceeds
zero. Then for each query Q ∈ Qorig we generate f, c+ and
c− by first generating two random numbers r1, r2 ∼ U[0, 1]
and then setting fQ = r1, c+
Q = r2 and c−
Q = 1 − r2. We
then generate the default workload Qdef as described in
Section 3.6 to achieve robustness to changes in Qorig. The
resulting workload is Q = Qorig ∪ Qdef.
Quality Metrics. We report the true (real) qual-
ity metrics reached by various techniques as defined in
Eq. (4) and (9). The expected quality metrics are used by
the algorithms only internally and thus not reported. As
common for these types of experiments, all techniques are
first applied to the probabilistic data without giving them
the knowledge of the ground truth tags. Then their out-
put is compared against the ground truth tags, which are
known for the dataset.
Approaches. We test the results of the following
approaches.
1. Branch and Bound (BB). Section 3.3.
2. Iterative Algorithm. Section 4.3.
3. Enum. Enumeration based approach.
Fig. 10. Cost of different approaches.
4. Thresholding. This approach is described in
Section 2.3. To set the threshold τ for the approach, we
randomly set aside 2/5 of the data as validation dataset
Dval and 3/5 as testing dataset Dtest. We pick the τ that
leads to minimum cost on Dval. All methods are compared
on Dtest. In XOR model, we use top-1 method to pick the
tag with maximum probability. For clarity of presentation,
we still call it as Thresholding.
5. Random guessing. To have an idea of which score
a mediocre approach would get, we use the following ran-
dom strategy as another baseline. For each tag wi ∈ W, the
random strategy generates a random number ri ∼ U[0, 1]
and includes wi into answer set A iff ri > 0.5.
6.2 Experiments
Experiment 1.•(Quality of various approaches). This exper-
iment compares all approaches in terms of the cost defined in
Eq. (4). The lower the cost is, the better the quality is.
Fig. 10 plots the cost of different approaches on three
datasets. As expected, on all three datasets, we can observe
that BB quickly achieves almost the same quality as that of
Enum for small value of Nleaf. We can notice that both Enum
and BB show about 30% improvement in cost over the baseline
threshold-based approach.
Experiment 2.•(Comparing to the Probabilistic-DBMS
solution). In the setting of our problem, object tags must
be stored as a deterministic representation. But a question
arises of how the cost/quality will be affected if this was not
a requirement. This experiment tests what would happen if
the existing applications, such as Flickr, start to allow proba-
bilistic input and also would be able to correctly interpret the
probabilities internally.
We refer to this solution as probabilistic solution Prob.
In this solution, there is no need to decide which tag should
be in the answer set. The probabilistic representation of object
O is kept in a probabilistic DB. Thus, Prob can opti-
mize over each individual query independently and choose
deterministic representation that leads to minimum expected
cost min(c+
Q · (1 − PO∈GQ ), c−
Q · PO∈GQ ). This approach pro-
vides us a sense of the best performance from the expected
perspective.
Notice, however, even though Prob could reach high qual-
ity, naturally Prob cannot be used directly for the problem
of determinzation studied in this paper. Hence the compari-
son is inherently unfair to determinization solutions, but we
nevertheless plot the Prob solution in all experiments to pro-
vide a sense of how close / far the determinization solution

Fig. 11. Accuracy of BB.
Fig. 12. Speedup of BB.
is compared to the best possible performance as would be the
case with the Prob approach.
Fig. 10 compares BB and Enum to the probabilistic solu-
tion. One interesting observation is that Enum and the BB
algorithm actually have similar quality to that of the proba-
bilistic solution, which is supposed to achieve the best quality
in terms of the expected cost.
Thus, interestingly, for the studied problem setting, stor-
ing data in a deterministic DB by applying determinization
procedure turns out to be a competitive solution (compared to
using probabilistic DBs), especially given that deterministic
DBs tend to be faster and more widespread.
Experiment 3.•(Performance of the BB algorithm). The goal
of the Branch and Bound algorithm is to reduce the expo-
nential search space of the problem, while maintaining almost
the same quality. The BB algorithm stops after observing Nleaf
answer sets. This experiment compares Enum and the BB algo-
rithm for different values of Nleaf in terms of both efficiency
and quality.
Fig. 11 shows the accuracy of the BB algorithm on
RealSpeech and LargeImg dataset, that is the fraction
of objects where the cost of answer set A generated by the BB
algorithm is equal to that of Aopt generated by Enum. The
accuracy of 1 means the cost of A always equal to the cost
of A. The accuracy shown in the figure is averaged over 50
different instances of query workload, where queries in each
instance are assigned random weights, false positive and false
negative costs. Fig. 12 shows the corresponding speedup by
the BB algorithm over the Enum, which is computed as the
time of Enum divided by the time of the BB algorithm.
On RealSpeech dataset, we can observe that after
expanding only Nleaf = 30 leaf nodes, the accuracy is above
0.9, while BB is still about 1000 times faster than Enum and
takes 0.06 second for the dataset. Similarly, on LargeImg
dataset, where objects are represented using the XOR model,
the BB algorithm quickly achieves the accuracy of 0.95 even
for small Nleaf = 15. The difference is that the speedup is now
200× and BB takes 1.43 second, while on the RealSpeech
dataset the speedup is above 2000× for Nleaf = 15. This, how-
ever, is as expected, since the average number of tags per object
Fig. 13. Speedup over non-optimized BB.
Fig. 14. Performance on special workloads.
in LargeImg is less than that in RealSpeech. The advan-
tage of BB is less in such cases.
Fig. 13 shows the improvement achieved by using query-
level optimization, as described in Section 3.4. When Nleaf is
small, the BB algorithm with query-level optimization shows
a 3× speedup over that without optimization, and the speedup
increases as Nleaf increases.
Experiment 4.•(Performance on various workloads). This
experiment tests different approaches on some special work-
loads: (a) uniform weights: weights of queries in the work-
load are the same. (b) equal costs: false positive cost is equal
to false negative cost for each query. (c) zipf’s weights:
weights of queries in the workload follow zipf’s distribu-
tion. As shown in Fig. 14, both Enum and BB outperforms
Thresholding across all different workloads. The gap
between BB and Thresholding in equal costs workload is
relatively small. This is as expected, because Thresholding
prefers consistent ratio of false positive and false negative costs
across different queries, while BB can adapt well to different
ratios.
Experiment 5 (Adaptivity to change in query workload).
The query workload Q provided for generating the determin-
istic representation, and the actual query workload Qact used
during querying can be different in practice. This experiment
tests how various approaches are affected as the original
workload Q is gradually changed into a completely different
workload Qrnd.
In this experiment, all the techniques are trained on
Q = Qorig ∪ Qdef to determinize uncertain objects. We
create another query workload Qrnd, which is the same as
Q, except the weights and costs of all queries in Qrnd are
generated as follows: for each query Q ∈ Qrnd, we first
generate two random numbers x1, x2 ∼ U[0, 1] and then
set fQ = x1, c+
Q = x2 and c−
Q = 1 − x2. In this way, some
queries such as those in Qdef, which is not important in Q,
may become important in Qrnd. On the other hand, those
queries with high weights in Q may become less important
in Qrnd.

Fig. 15. Adaptiveness.
Fig. 16. F1 of various methods.
The actual workload Qact is generated as a mix of Q
and Qrnd as follows. For each query Q1 ∈ Q and its corre-
sponding query Q2 ∈ Qrnd, we construct query Q ∈ Qact
as: (1) TQ = TQ1 = TQ2 ; (2) fQ = (1 − α)fQ1 + αfQ2 ; (3)
c−
Q = (1 − α)c−
Q1
+ αc−
Q2
; and (4) c+
Q = (1 − α)c+
Q1
+ αc+
Q2
,
where parameter α ∈ [0, 1] controls the relative contribution
of the Q and Qrnd workloads.
Fig. 15 shows the performance of the proposed BB algo-
rithm and the probabilistic solution on LargeImg dataset.
The actual query workload Qact changes from the original
workload Q (α = 0) into a different workload Qrnd (α = 1),
as α is changing from 0 to 1. The performance of BB remains
very close to that of Prob for small values of α (< 0.4).
Although not shown in the figure, BB is always better
than thresholding until the query workload becomes
completely different (α = 1). This figure illustrates that
the proposed BB approach is quite robust to small query
changes.
Experiment 6.•(F1 measure of various approaches). To
test the proposed iterative algorithm, we compare it with
other query unaware approaches, in terms of F1 measure.
Fig. 16 shows F1 measure of various approaches on both
RealSpeech and LargeImg datasets. We use object selec-
tion techniques in [5] as the probabilistic solution. It can
be seen that our iterative algorithm is better than
threshold-based algorithm. Enum approach is not involved
in comparison, because it is infeasible to test it even on a
small dataset. However, the quality of iterative algo-
rithm is very close to that of prob, which represents the
upper bound on quality achieved by any determinization
algorithm.
Experiment 7 (Efficiency of iterative algorithm). Fig. 17
tests the effect of data size on the efficiency of iterative
algorithm on LargeImg dataset. The number of objects in
LargeImg is changed from 1, 000 to 11, 000. The figure illus-
trates that time spent by iterative is almost linear in the
size of dataset.
Fig. 17. Iterative algorithm efficiency.
7 RELATED WORK
Determinizing Probabilistic Data. While we are not aware
of any prior work that directly addresses the issue of deter-
minizing probabilistic data as studied in this paper, the
works that are very related to ours are [5], [6]. They explore
how to determinize answers to a query over a probabilistic
database. In contrast, we are interested in best determin-
istic representation of data (and not that of a answer to a
query) so as to continue to use existing end-applications
that take only deterministic input. The differences in the
two problem settings lead to different challenges. Authors
in [13] address a problem that chooses the set of uncertain
objects to be cleaned, in order to achieve the best improve-
ment in the quality of query answers. However, their goal
is to improve quality of single query, while ours is to opti-
mize quality of overall query workload. Also, they focus on
how to select the best set of objects and each selected object
is cleaned by human clarification, whereas we determinize
all objects automatically. These differences essentially lead
to different optimization challenges. Another related area is
MAP inference in graphical model [14], [15], which aims to
find the assignment to each variable that jointly maximizes
the probability defined by the model. The determinization
problem for the cost-based metric can be potentially viewed
as an instance of MAP inference problem. If we view the
problem that way, the challenge becomes that of developing
fast and high-quality approximate algorithm for solving the
corresponding NP-hard problem. Section 3.3 exactly pro-
vides such algorithms, heavily optimized and tuned to
specifically our problem setting.
Probabilistic Data Models. A variety of advanced prob-
abilistic data models [16]–[18] have been proposed in the
past. Our focus however was determinizing probabilistic
objects, such as image tags and speech output, for which
the probabilistic attribute model suffices. We note that
determining probabilistic data stored in more advanced
probabilistic models such as And/Xor tree [6] might also be
interesting. Extending our work to deal with data of such
complexity remains an interesting future direction of work.
Index term selection. There are several related research
efforts that deal with the problem of selecting terms to
index document for document retrieval. A term-centric
pruning method described in [19] retains top postings for
each term according to the individual score impact that
each posting would have if the term appeared in an ad-
hoc search query. Authors in [20] propose a scalable term
selection for text categorization, which is based on cov-
erage of the terms. The focus of these research efforts is
on relevance – that is, getting the right set of terms that

are most relevant to document. In our problem, a set of
possibly relevant terms and their relevance to the docu-
ment are already given by other data processing techniques.
Thus, our goal is not to explore the relevance of terms to
documents, but to select keywords from the given set of
terms to represent the document, such that the quality of
answers to triggers/queries is optimized.
Query intent disambiguation. History data such as a
query workload and click graph have been used in the
research area of query intent disambiguation [21]–[24].
Query logs in these works are used to predict more relevant
terms for queries, or in other words more accurate intent
of queries. However, our goal is not to predict appropriate
terms, but to get the right keywords from the terms that
are already generated by automated data generation tool.
Query workload in our setting is not a source of feature for
classification, but used to drive the optimization of the end
performance of application. Thus techniques in these works
are not directly applicable in the context of the problem
domain we are addressing.
Query and tag recommendation. Another related
research area is that of query recommendation and tag
recommendation [25]–[28]. Based on a query-flow graph
representation of query logs, authors in [25] develop a mea-
sure of semantic similarity between queries, which is used
for the task of producing diverse and useful recommenda-
tions. Rae et al. [27] propose an extendable framework of
tag recommendation, using co-occurrence analysis of tags
used in both user specific context (personal, social con-
tact, social group) and non user specific context (collective).
The focus of these works is on how to model similari-
ties and correlations between queries/tags and recommend
queries/tags based on those information. However, our
goal is not to measure similarity between object tags and
queries, but to select tags from a given set of uncertain tags
to optimize certain quality metric of answers to multiple
queries.
8 CONCLUSION AND FUTURE WORK
In this paper we have considered the problem of deter-
minizing uncertain objects to enable such data to be stored
in pre-existing systems, such as Flickr, that take only
deterministic input. The goal is to generate a determinis-
tic representation that optimize the quality of answers to
queries/triggers that execute over the deterministic data
representation. We have proposed efficient determinization
algorithms that are orders of magnitude faster than the enu-
meration based optimal solution but achieves almost the
same quality as the optimal solution. As future work, we
plan to explore determinization techniques in the context of
applications, wherein users are also interested in retrieving
objects in a ranked order.
ACKNOWLEDGMENTS
This work is part of the UCI Sherlock Project
(http://sherlock.ics.uci.edu) on data quality and is
funded in part by NSF grants IIS-1118114, CNS 1063596,
CNS-1059436 and by DHS grant 206-000070.
REFERENCES
[1] D. V. Kalashnikov, S. Mehrotra, J. Xu, and
N. Venkatasubramanian, “A semantics-based approach for
speech annotation of images,” IEEE Trans. Knowl. Data Eng.,
vol. 23, no. 9, pp. 1373–1387, Sept. 2011.
[2] J. Li and J. Wang, “Automatic linguistic indexing of pictures by
a statistical modeling approach,” IEEE Trans. Pattern Anal. Mach.
Intell., vol. 25, no. 9, pp. 1075–1088, Sept. 2003.
[3] C. Wangand, F. Jing, L. Zhang, and H. Zhang, “Image annotation
refinement using random walk with restarts,” in Proc. 14th Annu.
ACM Int. Conf. Multimedia, New York, NY, USA, 2006.
[4] B. Minescu, G. Damnati, F. Bechet, and R. de Mori, “Conditional
use of word lattices, confusion networks and 1-best string
hypotheses in a sequential interpretation strategy,” in Proc.
ICASSP, 2007.
[5] R. Nuray-Turan, D. V. Kalashnikov, S. Mehrotra, and Y. Yu,
“Attribute and object selection queries on objects with probabilis-
tic attributes,” ACM Trans. Database Syst., vol. 37, no. 1, Article 3,
Feb. 2012.
[6] J. Li and A. Deshpande, “Consensus answers for queries over
probabilistic databases,” in Proc. 28th ACM SIGMOD-SIGACT-
SIGART Symp. PODS, New York, NY, USA, 2009.
[7] M. B. Ebarhimi and A. A. Ghorbani, “A novel approach for
frequent phrase mining in web search engine query streams,”
in Proc. 5th Annu. Conf. CNSR, Frederlcton, NB, Canada,
2007.
[8] S. Bhatia, D. Majumdar, and P. Mitra, “Query suggestions in the
absence of query logs,” in Proc. 34th Int. ACM SIGIR, Beijing,
China, 2011.
[9] C. Manning and H. Schutze, Foundations of Statistical Natural
Language Processing, Cambridge, MA, USA: MIT Press, 1999.
[10] D. V. Kalashnikov and S. Mehrotra, “Domain-independent data
cleaning via analysis of entity-relationship graph,” ACM Trans.
Database Syst., vol. 31, no. 2, pp. 716–767, Jun. 2006.
[11] K. Schnaitter, S. Abiteboul, T. Milo, and N. Polyzotis, “On-line
index selection for shifting workloads,” in Proc. IEEE 23rd Int.
Conf. Data Eng. Workshop, Istanbul, Turkey, 2007.
[12] P. Unterbrunner, G. Giannikis, G. Alonso, D. Fauser, and
D. Kossmann, “Predictable performance for unpredictable work-
loads,” in Proc. VLDB, Lyon, France, 2009.
[13] R. Cheng, J. Chen, and X. Xie, “Cleaning uncertain data with
quality guarantees,” in Proc. VLDB, Auckland, New Zealand,
2008.
[14] V. Jojic, S. Gould, and D. Koller, “Accelerated dual decomposition
for MAP inference,” in Proc. 27th ICML, Haifa, Israel, 2010.
[15] D. Sontag, D. K. Choe, and Y. Li, “Efficiently searching for
frustrated cycles in map inference,” in Proc. 28th Conf. UAI,
2012.
[16] P. Andritsos, A. Fuxman, and R. J. Miller, “Clean answers over
dirty databases: A probabilistic approach,” in Proc. 22nd ICDE,
Washington, DC, USA, 2006.
[17] O. Benjelloun, A. D. Sarma, and A. H. J. Widom, “ULDBs:
Databases with uncertainty and lineage,” in Proc. 32nd Int. Conf.
VLDB, Seoul, Korea, 2006.
[18] L. Antova, T. Jansen, C. Koch, and D. Olteanu, “Fast and simple
relational processing of uncertain data,” in Proc. IEEE 24th ICDE,
Cancun, Mexico, 2008.
[19] D. Carmel et al., “Static index pruning for information retrieval
systems,” in Proc. 24th Annu. Int. ACM SIGIR, New Orleans, LA,
USA, 2001.
[20] J. Li and M. Sun, “Scalable term selection for text categorization,”
in Proc. EMNLP-CoNLL, Prague, Czech Republic, 2007.
[21] X. Li, Y. Wang, and A. Acero, “Learning query intent from
regularized click graphs,” in Proc. 31st Annu. Int. ACM SIGIR,
Singapore, 2008.
[22] E. Pitler and K. Church, “Using word-sense disambiguation meth-
ods to classify web queries by intent,” in Proc. Conf. EMNLP,
Singapore, 2009.
[23] A. Ashkan, C. L. Clarke, E. Agichtein, and Q. Guo, “Classifying
and characterizing query intent,” in Proc. 31th ECIR, Toulouse,
France, 2009.
[24] J. Teevan, S. Dumais, and D. Liebling, “To personalize or not to
personalize: Modeling queries with variation in user intent,” in
Proc. SIGIR, Singapore, 2008.
[25] I. Bordino, C. Castillo, D. Donato, and A. Gionis, “Query similar-
ity by projecting the query-flow graph,” in Proc. 33rd Int. ACM
SIGIR, Geneva, Switzerland, 2010.

[26] A. Anagnostopoulos, L. Becchetti, C. Castillo, and A. Gionis, “An
optimization framework for query recommendation,” in Proc. 3rd
ACM Int. Conf. WSDM, New York, NY, USA, 2010.
[27] A. Rae, B. Sigurbjörnsson, and R. V. Zwol, “Improving tag
recommendation using social networks,” in Proc. RIAO, Paris,
France, 2010.
[28] B. Sigurbjörnsson and R. V. Zwol, “Flickr tag recommendation
based on collective knowledge,” in Proc. 17th Int. Conf. WWW,
New York, NY, USA, 2008.
Jie Xu received the BSc degree in computer sci-
ence from Zhejiang University, China, in 2006
and the MSc degree in computer science from
Zhejiang University, China, in 2008. He is cur-
rently a PhD candidate in the Computer Science
Department, University of California, Irvine, CA,
USA. His current research interests include infor-
mation retrieval, machine learning, and computer
vision.
Dmitri V. Kalashnikov is an associate adjunct
professor of computer science at the University
of California, Irvine, CA, USA. He received the
PhD degree in computer science from Purdue
University in 2003. He received the diploma in
applied mathematics and computer science from
Moscow State University, Russia, in 1999, grad-
uating summa cum laude. His current research
interests include databases and data mining.
Currently, he specializes in the areas of entity
resolution & data quality, and real-time situational
awareness. In the past, he has also contributed to the areas of spa-
tial, moving-object, and probabilistic databases. He has received several
scholarships, awards, and honors, including an Intel Fellowship and Intel
Scholarship. His work is supported by the US NSF, DH&S, and DARPA.
Sharad Mehrotra received the PhD degree in
computer science from the University of Texas
at Austin in 1993. He is currently a professor
in the Department of Computer Science at the
University of California, Irvine, CA, USA (UCI)
and the director of the Center for Emergency
Response Technologies. Previously, he was a
professor at the University of Illinois at Urbana-
Champaign (UIUC). He has received numerous
awards and honors, including SIGMOD Best
Paper Award 2001, DASFAA Best Paper Award
2004, and CAREER Award 1998 from the US National Science
Foundation (NSF). His current research interests include area of
database management, distributed systems, and data analysis. He is
a member of the IEEE.
For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/publications/dlib.

Query Aware Determinization of Uncertain Objects

More Related Content

What's hot (20)

Viewers also liked (16)

Similar to Query Aware Determinization of Uncertain Objects (20)

Recently uploaded (20)

Query Aware Determinization of Uncertain Objects