Accessing the Insights Page
You can access the assessment’s insights by clicking on the Insights tab of an assessment, as shown in the image below.
Terminology
The Insights page uses the following terminology:
Assessment Rank: For every assessment, all the candidates are ranked based on the assessment score.
Assessment Score : A comprehensive score which is the function of Success (challenges successfully solved), Efficiency(scoring points with least number of tries), and Speed(quickness in completing the assessment).
This is a number score between 0 and 100.
Success: Success is measured as a number score between 0-100.
Success is defined by the points earned by successfully solving challenges where: 0= No points achieved; 100= All Points achieved
Efficiency: Efficiency is measured as a number score between 0-100.
Efficiency is the individual's ability to score points in least number of tries where 0= All tasks entered but not solved; 100= All tasks entered are solved
Speed: Speed is measured as a number score between 0-100.
Speed is determined by the individual's capability to complete the assessment in the quickest possible time where 0= 0 minutes spent, and 100= Assessment duration
Score of a skill: Like assessment score, score in a skill is calculated as a function of
Success in solving challenges that are assessing for that skill.
Efficiency in solving challenges that are assessing for that skill. This is measured by the ability to solve with least number of tries.
Speed in solving challenges that are assessing for that skill. This is measured by quickness in solving these challenges.
Professional Specialty-Professional specialties denote broad categories of cyber operations. Each challenge aligns with one of these specialties, providing a structured framework for classification.
Skills- Skills indicate the specific, acquired ability necessary to solve a challenge within a given timeframe or effort. Each challenge should necessitate one or two specific skills linked to the relevant professional specialty.
Techniques- Technologies include the environments (e.g., Windows, Linux, Docker) and tools (e.g., Splunk, Nessus, Python) incorporated within the challenge, crucial for its resolution.
Technologies-Techniques refer to the specific methods or strategies required to resolve a challenge. There is a preset list of techniques to select from, with the option to add more if needed.
Viewing Insights
You can select which participant’s insights you would like to view by clicking on the Select Candidate[1] dropdown and choosing the participant from the Participants [2] menu, as shown in the image below. You can also be directed to this page, if you selected to see detailed performance and approach of a particular candidate on the Assessment Overview page.
The Insights page is divided into three sections:
Overall Assessment performance : This section gives you an overall view of how successful, efficient and quick the candidate has been in this assessment
Performance within skills: This section gives you a view of how successful, efficient and quick the candidate has been within each skill that was assessed for in this assessment.
Approach: This section captures the candidate's allocation of time across each attempted challenge, number of attempts and whether or not they were able to solve it, providing a nuanced perspective on their approach.
Section 1: Overall Assessment performance
You can see the Assessment Rank[1] of the candidate on the top left. The assessment rank is dependent on on the Assessment Score[2] which constitutes three metrics:
Success[3]
Efficiency[4]
Speed[5]
Reading the distribution bar
In the distribution bars, the light coloured part in the base represents the range covering lowest and highest scores of the cohort.
The darker region represents one standard deviation above and below the cohort average. Simply put, a wider shaded region indicates that the scores of the cohort are more spread out from the average. Conversely, a smaller shaded region suggests that the majority of scores are clustered closely around the average.
The dark line on the bar shows where the candidate's score is for that measure.
Performance Triangle
Performance triangle gives a glimpse of how the candidate performed overall in the assessment. The success of the performance is determined by the challenges successfully solved as well as their efficiency and speed in solving those tasks.
The dark grey line line shows the cohort average, and the blue line shows the candidate score.
Section 2: Performance within Skills, Technologies and Techniques
Skill performance compared to cohort
This section demonstrates the performance of a participant in different skills that are being assessed in this assessment. The skills are plotted on the x-axis and named in their abbreviations. On y-axis, is the skill score which is a score calculated based on the success, efficiency and speed demonstrated in solving challenges within that skill.
The blue line shows their score in a particular skill, and the black line shows the average performance of the cohort. You can hover over the line to see the performance in that skill compared to the cohort average.
Techniques and Technologies
The bar chart shows the techniques and technologies in this assessment. The green area shows successful demonstration by the candidate for that technique or technology.
Section 3: Approach
This visualization offers valuable insights into the candidate's journey, shedding light on the candidate's endeavor by showcasing a chronological display of when they viewed a challenge, when they entered it, when and how many times they attempted it and when they solved it. This captures the candidate's allocation of time across each attempted challenge, providing a nuanced perspective on their approach.
The x-axis represents time, and each row represents a challenge that is included in the assessment.
Following candidate activity along with the time is recorded on this table:
View- A “View” event is when a candidate opens a challenge and views its basic details.
Started- After having viewed the basic details, an “Start” event occurs when a candidate engages further transitioning into the expanded challenge view. The expanded challenge view encompasses attachments, lab or container environments, and hints.
Attempt- An "Attempt" is logged when a candidate submits their response or FLAG for the challenge. Candidates have the flexibility to make multiple attempts according to their preference. Nonetheless, their efficiency rating gradually diminishes with each subsequent attempt.
Solved-A challenge is marked “Solved” when the candidate enters the correct FLAG
Each of these actions are represented by an icon as following:
The icons appear in the timeline[1] representing the activity aligned to the time of occurrence. You can click view[2] for comprehensive challenge details, including valuable insights from the candidate's Notes, providing a glimpse into their problem-solving approach and thought process.
The Challenge modal gives the status[1] of the challenge related activity (whether it has been viewed, started, attempted, or solved), number of attempts[2], Professional Specialty[3] the challenge belongs to and the Proficiency level[4] in the top bar.
You can also read the challenge description[5], Skills[6] that are being assessed in this challenge. The number in parenthesis next to the skill title indicates the number of skills.
You can also see the techniques and technologies[7] included in this assessment. In the bar graph technique/technology , the light grey bar represents the weightage of that technique/technology in the entire assessment. The green bar/dark grey bars indicates the weightage of that technique/technology offered in this challenge, where the green means that the technique/technology has been successfully achieved, and the dark grey means that it has not been achieved.
Most importantly you can see the Candidate’s Notes[8] to understand their approach better. This is helpful in making informed evaluation regarding a candidate’s thought process. To close the modal, click Close[9].
Downloading Assessment Insights PDF report
To download these assessment insights for a candidate in a PDF report format, simply click the download button on the top right.
