Modeling Factual Claims with Frames

In this paper we introduce an extension of FrameNet for structured and semantic modeling of factual claims and an adaptation of the frame detection algorithms in Open Sesame for identifying frames and extracting frame elements from text. This claim modeling capability can be leveraged in assisting a variety of steps for automating fact-checking, e.g., matching claims with fact-checks, translating claims to structured queries, and so on. Our preliminary results show that while many challenges remain, which we discuss, frames can potentially improve the aforementioned steps. Further studies will reveal the strength and weakness of this modeling approach in more detail, as well as how to incorporate it into the full pipeline of fact-checking automation.

1. Modeling)Factual)Claims)with)Frames
The$Innovative$Database$and$Information$Systems$Research$Laboratory$(IDIR)
Fatma%Arslan,%Damian%Jimenez,%Josue%Caraballo,%Gensheng%Zhang,%Chengkai%Li
✻Agrowinginterestinautomatingvariousfact!
checkingsteps.
✻WeintroduceanextensionofFrameNet(Baker
et.al.,1998)forstructuredandsemantic
modelingoffactualclaims.
✻Withsuchmodelingcapability,wecancapture
variousaspectsofafactualclaim;thedomain
andtopicoftheclaim,theentitiesinvolvedand
theirrelationships,quantities,comparisons,
aggregatestructures,andsoon.
✻Thisclaimmodelingcapabilityisusefulfora
varietyofstepsforautomatingfact!checking,
e.g.,detectingfactualclaims,matchingclaims
withfact!checks,translatingclaimsto
structuredqueries,andsoon.
Introduction
Figure:VoteandOccupyrankframesfromourframecollectionalong
withtheirLUs.AnexemplifiedsentenceforvoteandrankLuswith
color!codedframeannotationsbelow.Labelsbeloweachsentenceare
framespecificframeelements(FEs).
Modeling)Factual)Claims
Preliminary*Experiments:*Claim*Detection
Defining'New'Frames
Table:Framepredictionperformance,intermsofPrecision(P),Recall(R)andF!measure(F1).Whereavg
w
denotesthe
weightedaverageofcorrespondingmeasureacrosstenframes.
Conclusions)and)Future)Work
Task:Identifyingfactualclaimstobefact!checked.
✻Utilizedopen!sesame
2
forframeidentification.
✻RetraineditonFrameNet1.7datasetalongwithannotated
sentencesforthreenewframes.
1.#https://www.politifact.com#2.#https://github.com/swabhs/open!sesame%%%3.%http:%//www.sharethefacts.org/%
✻Labelledfactualclaimsfrom“SharetheFacts”dataset
3
for
10frames.
✻Evaluatedopen!sesame’sperformanceforclaimdetection.
✻Collectedfact!checkedclaimsfromPolitiFact
1
,
✻Examinedasubsetoftheseclaimsonebyone.
✻Groupedclaimssimilarinnature.
✻Creatednewframesforeachclaimgroupifit
doesn’talreadyexistinFrameNetcollection.
✻20frames(13newlycreated)
✻900labelledfactualclaims.
✻UniquenessoftraitandVoteframes
performedinlinewithotherpre!
establishedframes.
✻Model’sperformancecanbeimproved
byfeedingmoretrainingdata.
✻Developedaframeannotatortoolto
annotatemoredataforfurther
experiments.
Oppose&and&Support&Consistency&Frame
ThisresearchhasbeenpartiallysupportedbyNationalScienceFoundation
grant#1719054andasubawardfromtheDukeTech&CheckCooperative
(fundedbyTheKnightFoundation,theFacebookJournalismProjectand
theCraigNewmarkFoundation).Anyopinions,findings,andconclusions
orrecommendationsexpressedinthispublicationarethoseofthe
authorsanddonotnecessarilyreflecttheviewsofthefundingagencies.
Funding: