Class of active learning functions More...

Public Member Functions
	ActiveLearning (IList< Datum > data, BCC model, Results results, int numCommunities)
	Constructs an active learning instance with a specified data set and model instance. More...

void	UpdateActiveLearningResults (Results results)
	Updates the active learning results object. More...

Dictionary< string, ActiveLearningResult >	EntropyTrueLabel ()
	Computes the entropy on the true label posterior distribution of the active learning results. More...

Static Public Member Functions
static void	RunActiveLearning (IList< Datum > data, string modelName, RunType runType, BCC model, TaskSelectionMethod taskSelectionMethod, WorkerSelectionMethod workerSelectionMethod, string resultsDir, int communityCount=-1, int initialNumLabelsPerTask=1, int numIncremData=1)
	Runs the standard active learning procedure on a model instance and an input data set. More...

static void	RunParallelActiveLearning (IList< Datum > data, string[] modelName, RunType[] runType, BCC[] model, TaskSelectionMethod[] taskSelectionMethod, WorkerSelectionMethod[] workerSelectionMethod, int communityCount=-1, int initialNumLabelsPerTask=1, int numIncremData=1)
	Runs the standard active learning procedure in parallel on an array of model instances and an input data set. More...

static void	DoSnapshot (List< double > accuracy, List< double > avgRecall, List< ActiveLearningResult > taskValue, Results results, string modelName, string suffix, string resultsDir, int projectInitialNumLabelsPerTask, double lipschitzConstant=-1)
	Saves the results of the inference and the model's parameters on csv files. More...

static List< Datum >	GetSubdata (Dictionary< string, Datum[]> groupedRandomisedData, Dictionary< string, int > currentCounts, Dictionary< string, HashSet< string >> workersPerTask)
	Returns a list of sub-data selected sequentially from the input data list. More...

static List< Datum >	GetNextData (Dictionary< string, Datum[]> groupedRandomisedData, List< Tuple< string, string, ActiveLearningResult >> labelValue, Dictionary< string, int > currentCounts, Dictionary< string, int > totalCounts, Dictionary< string, HashSet< string >> workersPerTask, int numIncremData)

static Datum	GetRandomDatum (Dictionary< string, Datum[]> groupedRandomisedData, Dictionary< string, int > currentCounts, Dictionary< string, HashSet< string >> workersPerTask)

static void	ResetAccuracyList ()

static void	ResetParallelAccuracyList (int numModels)

Static Public Attributes
static Boolean	isExperimentCompleted = false
	Indicate whether the experiment is completed or not, used for the GUI More...

static List< double >	accuracy
	Static accuracy list used for the gui More...

static List< double >[]	accuracyArray
	Static array of accuracy lists used for the gui More...

static List< ActiveLearningResult >	taskValueList
	List of utility values for the tasks selected at each round More...

static List< ActiveLearningResult >[]	taskValueListArray
	Array of lists of utility values for the tasks selected at each round in parallel mode More...

static Results[]	results
	List of Results for showing the confusion matrices of specific worker, made global variable for GUI More...

Detailed Description

Class of active learning functions

Constructor & Destructor Documentation

CrowdsourcingModels.ActiveLearning.ActiveLearning	(	IList< Datum >	data,
		BCC	model,
		Results	results,
		int	numCommunities
	)

inline

Constructs an active learning instance with a specified data set and model instance.

Parameters

data	The data.
model	The model instance.
results	The results instance.
numCommunities	The number of communities (only for CBCC).

Builds the full matrix of data from every task and worker

Member Function Documentation

static void CrowdsourcingModels.ActiveLearning.DoSnapshot	(	List< double >	accuracy,
		List< double >	avgRecall,
		List< ActiveLearningResult >	taskValue,
		Results	results,
		string	modelName,
		string	suffix,
		string	resultsDir,
		int	projectInitialNumLabelsPerTask,
		double	lipschitzConstant = `-1`
	)

inlinestatic

Saves the results of the inference and the model's parameters on csv files.

Parameters

accuracy	The list of accuracies evaluated on the gold labels at each active learning round.
avgRecall	The list of average recalls evaluated on the gold labels at each active learning round.
taskValue	The list of utilities of the task selected at each active learning round.
results	The result instance.
modelName	The model name.
suffix	The suffix of the csv files.
resultsDir	The directory to store the csv files.

Edit this print line to get the accuracy printed if the format that you want.

Dictionary<string, ActiveLearningResult> CrowdsourcingModels.ActiveLearning.EntropyTrueLabel ( )

inline

Computes the entropy on the true label posterior distribution of the active learning results.

Returns: A dictionary keyed by the TaskId and the value is the true label entropy.

static List<Datum> CrowdsourcingModels.ActiveLearning.GetSubdata	(	Dictionary< string, Datum[]>	groupedRandomisedData,
		Dictionary< string, int >	currentCounts,
		Dictionary< string, HashSet< string >>	workersPerTask
	)

inlinestatic

Returns a list of sub-data selected sequentially from the input data list.

Parameters

groupedRandomisedData	The randomised data.
currentCounts	The current data count per task.
workersPerTask	The dictionary keyed by taskId and the value is an hashset of workerId who have remaining labels for the tasks.

Returns: The list of sub-data.

static void CrowdsourcingModels.ActiveLearning.RunActiveLearning	(	IList< Datum >	data,
		string	modelName,
		RunType	runType,
		BCC	model,
		TaskSelectionMethod	taskSelectionMethod,
		WorkerSelectionMethod	workerSelectionMethod,
		string	resultsDir,
		int	communityCount = `-1`,
		int	initialNumLabelsPerTask = `1`,
		int	numIncremData = `1`
	)

inlinestatic

Runs the standard active learning procedure on a model instance and an input data set.

Parameters

data	The data.
modelName	The model name.
runType	The model run type.
model	The model instance.
taskSelectionMethod	The method for selecting tasks (Random / Entropy).
workerSelectionMethod	The method for selecting workers (only Random is implemented).
resultsDir	The directory to save the log files.
communityCount	The number of communities (only for CBCC).
initialNumLabelsPerTask	The initial number of exploratory labels that are randomly selected for each task.

We create a list of task utilities

We create a list of worker utilities

Create a list of tuples <TaskId, WorkerId, ActiveLearningResult>

static void CrowdsourcingModels.ActiveLearning.RunParallelActiveLearning	(	IList< Datum >	data,
		string[]	modelName,
		RunType[]	runType,
		BCC[]	model,
		TaskSelectionMethod[]	taskSelectionMethod,
		WorkerSelectionMethod[]	workerSelectionMethod,
		int	communityCount = `-1`,
		int	initialNumLabelsPerTask = `1`,
		int	numIncremData = `1`
	)

inlinestatic

Runs the standard active learning procedure in parallel on an array of model instances and an input data set.

Parameters

data	The data.
modelName	The model name.
runType	The model run type.
model	The model instance.
taskSelectionMethod	The method for selecting tasks (Random / Entropy).
workerSelectionMethod	The method for selecting workers (only Random is implemented).
resultsDir	The directory to save the log files.
communityCount	The number of communities (only for CBCC).
initialNumLabelsPerTask	The initial number of exploratory labels that are randomly selected for each task.

Main loop

Run through all the models

We create a list of worker utilities

Create a list of tuples <TaskId, WorkerId, ActiveLearningResult>

void CrowdsourcingModels.ActiveLearning.UpdateActiveLearningResults ( Results results )

inline

Updates the active learning results object.

Parameters

results The new results

Member Data Documentation

List<double> CrowdsourcingModels.ActiveLearning.accuracy

static

Static accuracy list used for the gui

List<double> [] CrowdsourcingModels.ActiveLearning.accuracyArray

static

Static array of accuracy lists used for the gui

Boolean CrowdsourcingModels.ActiveLearning.isExperimentCompleted = false

static

Indicate whether the experiment is completed or not, used for the GUI

Results [] CrowdsourcingModels.ActiveLearning.results

static

List of Results for showing the confusion matrices of specific worker, made global variable for GUI

List<ActiveLearningResult> CrowdsourcingModels.ActiveLearning.taskValueList

static

List of utility values for the tasks selected at each round

List<ActiveLearningResult> [] CrowdsourcingModels.ActiveLearning.taskValueListArray

static

Array of lists of utility values for the tasks selected at each round in parallel mode

The documentation for this class was generated from the following file:

C:/Users/Matteo/Source/Repos/active-crowd/CrowdsourcingModels/ActiveLearning.cs

Public Member Functions

Static Public Member Functions

Static Public Attributes

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation