Skip to main content

NAEP Technical DocumentationDevelopment of the Technology and Engineering Literacy Indices

For the 2014 technology and engineering literacy (TEL) assessment, several indices of policy interest were developed that satisfied both theoretical criteria based on content analysis, and empirical criteria based on multivariate statistical techniques. This resulted in the creation of several new reporting elements. The development of the 2014 TEL indices can be summarized in three main steps:

1. Question selection.  Because of the special nature of the TEL assessment, new contextual items such as those exploring technology learning and use were developed and included in the 2014 TEL student questionnaire with the intention of developing meaningful indices. A content analysis of the 2014 TEL survey questionnaires was then conducted to select sets of the questions that were theoretically interpretable and meaningful as a conceptual unit, as potential indices to measure specific constructs of interest.

2.  Examination of empirical relationships. Factor analysis was used to explore and verify the empirical properties of the data. Construct validity of the potential indices was evaluated through factorial validity with respect to the survey question responses, and the convergent and discriminant validity of the factor with respect to other factors. If the factor had the expected pattern of relationships and non-relationships, the construct validity of the factor as representing the intended index was supported.

3. Index scoring. The partial credit item response theory (IRT) model was used to scale the indices. From the IRT model, expected a priori (EAP) scores were generated. Then, the EAP scores were transformed to have a mean of 10 and a standard deviation of 2, and were reported on a scale from 0 – 20. Students with incomplete data on the index were used to estimate the parameters of the IRT model, but were not assigned an EAP score.

Index of TEL Confidence

The table below presents the items forming the index of students’ TEL confidence. This index was designed to measure students’ belief in their abilities to do various technology and engineering-related tasks. For each of eight questions, students were asked to indicate their confidence in doing the task described in the question (I definitely can’t, I probably can’t, maybe, I probably can, or I definitely can).


Items in the index of eighth-grade students' TEL confidence, grade 8 technology and engineering literacy national assessment: 2014
Do you think that you would be able to do each of the following? Select one circle in each row.
Response categories                                 
ItemI definitely can'tI probably can'tMaybeI probably canI definitely can
NOTE: In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.
D812201a.Create presentations with sound, pictures, or videoABCDE
D812202b.Organize information into a chart, graph, or spreadsheetABCDE
D812203c.Compare products using the InternetABCDE
D812204d.Use tools or materials to fix somethingABCDE
D812205e.Take something apart in order to fix it or see how it worksABCDE
D812206f.Describe how inventions change societyABCDE
D812207g.Compare how different activities affect the environmentABCDE
D812208h.Explain why people have different tools, machines, or devices in different parts of the worldABCDE

Indices of Technology and Society Related Learning

Two indices were designed to measure students’ learning of technology and society related topics. For the first index, students responded to six questions related to the frequency with which they learned about technology and society related topics in school (never, rarely, sometimes, or often). For the second index, students responded to four questions related to the frequency with which they learned about technology and society related topics outside of school (never, rarely, sometimes, or often). The tables below present the items making up the index of students’ in-school learning and outside-of-school learning, respectively.

Items in the index of eighth-grade students' technology and society related learning in school, grade 8 technology and engineering literacy national assessment: 2014
In school, how often have you learned about or discussed the following? Select one circle in each row.
Response categories                
ItemNeverRarelySometimesOften
NOTE: In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.
D805201a.Inventions that change the way people liveABCD
D805301b.Choices people make that affect the environmentABCD
D805401c.Conditions that influence the use or availability of machines or devicesABCD
D805501d.The ways people work together to solve problems in their community or the worldABCD
Items in the index of eighth-grade students' technology and society related learning in school, grade 8 technology and engineering literacy national assessment: 2014
In school, how often have you learned about or discussed the following? Select one circle in each row.
Response categories                
ItemNeverRarelySometimesOften
D812001a.How to judge reliability of sources (for example, how a website might be biased or inaccurate)ABCD
D812101b.How to credit others for their ideas (for example, citing sources, using endnotes and footnotes in reports)ABCD
NOTE: In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

Items in the index of eighth-grade students' technology and society related learning outside of school, grade 8 technology and engineering literacy national assessment: 2014
Outside of school, how often have you learned about or discussed the following? Select one circle in each row.
Response categories                     
ItemNeverRarelySometimesOften
NOTE: In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.
D805601a.Inventions that change the way people liveABCD
D805701b.Choices people make that affect the environmentABCD
D805801c.Conditions that influence the use or availability of machines or devicesABCD
D805901d.The ways people work together to solve problems in their community or the worldABCD

Top

Indices of Design and Systems Related Learning

Two indices were designed to measure students’ engagement in design and systems related learning activities. For the first index, students responded to six questions related to the frequency with which they engaged in design and systems related learning activities at school. (See the two tables below for response categories for the six questions.)  For the second index, students responded to five questions related to the frequency with which they engaged in design and systems related learning activities outside of school (never, once or twice, three to five times, or more than five times). The tables below present the items forming the index of students’ in-school learning and outside-of-school learning, respectively.

Items in the index of eighth-grade students' design and systems related learning in school, grade 8 technology and engineering literacy national assessment: 2014
In school, how often have you ever done the following activities? Select one circle in each row.
Response categories                          
ItemNeverOnce or twiceThree to five timesMore than five times
NOTE: In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.
D809301a.Used different tools, materials, or machines to see which are best for a given purposeABCD
D809401b.Built or tested a model to see if it solves a problemABCD
D809601c.Figured out why something is not working in order to fix itABCD
D809501d.Taken something apart in order to fix it or see how it worksABCD
Items in the index of eighth-grade students' design and systems related learning in school, grade 8 technology and engineering literacy national assessment: 2014
In school, how often have you learned about or discussed the following? Select one circle in each row.
Response categories                          
ItemNeverRarelySometimesOften
D801601a.Designing or creating something to solve a problemABCD
D801701b.Designing something when there is limited time, money, or materialsABCD
NOTE: In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

Items in the index of eighth-grade students' design and systems related learning outside of school, grade 8 technology and engineering literacy national assessment: 2014
Outside of school, how often have you ever done the following activities? Select one circle in each row.
Response categories                          
ItemNeverOnce or twiceThree to five timesMore than five times
NOTE: In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.
D810001a.Used tools or materials to plan or design something (for example, cake recipe, party)ABCD
D810101b.Used different tools, materials, or machines to see which are best for a given purposeABCD
D810301c.Built or tested a model to see if it solves a problemABCD
D810201d.Figured out why something is not working in order to fix itABCD
D810401e.Taken something apart in order to fix it or see how it worksABCD

Index of Computer Use

The tables below present the items forming the index of students’ computer use. This index was designed to measure students’ use of computers or other digital technology in or outside of school. For each of six questions, students were asked to indicate the frequency with which they use a computer or other digital technology for various activities in or outside of school (never or almost never, a few times a year, once or twice a month, once or twice a week, or every day or almost every day).

Items in the index of eighth-grade students' computer use, grade 8 technology and engineering literacy national assessment: 2014
For school work, how often do you use a computer or other digital technology for the following activities? Select one circle in each row.
Response categories                                   
ItemNever or almost neverA few times a yearOnce or twice a monthOnce or twice a weekEvery day or almost every day
NOTE: In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.
D803701a.Create, edit, or organize digital mediaABCDE
D803901b.Create a presentationABCDE
D804001c.Create a spreadsheet (a table or grid that displays data into columns and rows and may be used to create charts and graphs)ABCDE
Items in the index of eighth-grade students' computer use, grade 8 technology and engineering literacy national assessment: 2014
In this question, please think about activities you do that are not related to your school work. How often do you use a computer or other digital technology for the following activities not for school work? Select one circle in each row.
Response categories                                   
ItemNever or almost neverA few times a yearOnce or twice a monthOnce or twice a weekEvery day or almost every day
D804301a.Create, edit, or organize digital mediaABCDE
D804501b.Create a presentationABCDE
D804601c.Create a spreadsheet (a table or grid that displays data into columns and rows and may be used to create charts and graphs)ABCDE
NOTE: In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

Top

Using EAP scores generated from the partial credit IRT model, student responses were converted to a reporting index ranging from 0 to 20, with a mean of 10 and a standard deviation of 2, as described above. The following tables show the IRT parameters for the 2014 TEL indices.

IRT parameters for the TEL confidence index, grade 8 technology and engineering literacy national assessment, by item: 2014
Item
bj dj1 dj2 dj3 dj4
D812201-0.980.33-0.74-0.27-0.80
D812202-0.980.72-0.54-0.30-0.96
D812203-1.060.630.55-0.34-0.83
D812204-0.920.82-0.63-0.38-1.07
D8122050.650.78-0.53-0.38-0.93
D812206-0.620.890.55-0.40-1.05
D812207-0.730.89-0.65-0.37-1.17
D812208-0.660.790.55-0.32-1.03
† Not applicable. Parameters are missing for several items in the table because categories were collapsed.            
NOTE: The number of dj parameters is one less than the number of categories for the item. In 2014, grades 4 and 12 were not assessed. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i; dji may not sum to zero because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

IRT parameters for the technology and society in-school learning index, grade 8 technology and engineering literacy national assessment, by item: 2014
Item
bj dj1 dj2 dj3 dj4
D812001
-0.170.830.29-1.12
D812101-0.610.570.17-0.74
D805201-0.510.880.25-1.13
D805301-0.591.040.20-1.25
D805401-0.151.140.15-1.29
D805501-0.530.920.17-1.09
† Not applicable.
NOTE: The number of dj parameters is one less than the number of categories for the item. In 2014, grades 4 and 12 were not assessed. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i; dji may not sum to zero because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

IRT parameters for the technology and society out-of-school learning index, grade 8 technology and engineering literacy national assessment, by item: 2014
Item
bj dj1 dj2 dj3 dj4
D8056010.001.120.03-1.15
D805701-0.191.100.09-1.19
D8058010.111.160.03-1.20
D805901-0.121.070.03-1.10
† Not applicable.  Parameters are missing for several items in the table because categories were collapsed.
NOTE: The number of dj parameters is one less than the number of categories for the item. In 2014, grades 4 and 12 were not assessed. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i; dji may not sum to zero because of rounding. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

IRT parameters for the design and systems in-school learning index, grade 8 technology and engineering literacy national assessment, by item: 2014
Item
bj dj1 dj2 dj3 dj4
D801601
-0.471.030.13-1.16
D8017010.051.020.10-1.11
D8093010.111.03-0.24-0.79
D8094010.280.96-0.19-0.77
D8095010.590.55-0.23-0.32
D8096010.150.81-0.18-0.63
† Not applicable.
NOTE: The number of dj parameters is one less than the number of categories for the item. In 2014, grades 4 and 12 were not assessed. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i; dji may not sum to zero because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

Top

IRT parameters for the design and systems out-of-school learning index, grade 8 technology and engineering literacy national assessment, by item: 2014
Item
bj dj1 dj2 dj3 dj4
D810101-0.080.85-0.20-0.66
D8103010.270.76-0.17-0.59
D810401-0.110.63-0.26-0.37
D810201-0.460.79-0.20-0.59
D810001-0.550.71-0.25-0.46
† Not applicable.  Some parameters are missing for item D810001 because categories were collapsed.
NOTE: The number of dj parameters is one less than the number of categories for the item. In 2014, grades 4 and 12 were not assessed. For items scaled with the partial credit model, parameters a and c are not estimated. The parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i; dji may not sum to zero because of rounding. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

IRT parameters for the computer use index, grade 8 technology and engineering literacy national assessment, by item: 2014
Item
bj dj1 dj2 dj3 dj4
D8037010.300.710.14-0.23-0.61
D8040010.610.910.14-0.30-0.75
D8039010.321.640.29-0.79-1.14
D8043010.450.310.11-0.09-0.33
D8046010.980.400.24-0.13-0.51
D8045010.900.690.31-0.30-0.70
† Not applicable. Some parameters are missing for item D804301 because categories were collapsed.            
NOTE: The number of dj parameters is one less than the number of categories for the item. In 2014, grades 4 and 12 were not assessed. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i; dji may not sum to zero because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

The tables below show the index values and transformed scale scores for each of the six TEL indices.

As a reporting aid, cut points were selected to divide the index scores into a range of categories or classifications. For example, for the index of technology and society related learning in school, index scores were divided into never, rarely, sometimes, and often classifications, such that students with high index scores (i.e., 11.8 or higher on a scale of 0 to 20) generally reported that they often learned about or discussed technology and society related topics as part of their school work. Students with low scores on the index (i.e., lower than 6.6) generally reported that they never learned about or discussed these topics at school.

For each item, response categories were coded as numerical values. The partial credit IRT model was used to scale the indices, and each index value corresponds to a response average. High response averages correspond to high index values, and low response averages correspond to low index values. The index scores were divided into a range of classifications based on the response averages. The cut points for each index were determined based on the average response to the relevant set of items. For instance, for the index of technology and society related learning in school, students were classified as follows:

  • Students with index scores associated with an average response less than 1.5 (index value -1.7114) were classified as never on the index. That is, students who on average responded closest to never on a question with the response options never, rarely, sometimes, and often were classified as never learning about or discussing technology and society related topics in school.
  • Students with index scores associated with an average response greater or equal to 1.5 but less than 2.5 (index value -0.5189) were classified as rarely learning about or discussing technology and society related topics in school.
  • Students with index scores associated with an average response greater or equal to 2.5 but less than 3.5 (index value 0.9134) were classified as sometimes on the index.
  • Finally, students with index scores associated with an average response greater or equal to 3.5, were classified as often on the index. That is, students who on average responded closest to often, were classified as often learning about or discussing technology and society related topics in school.
Index values and transformed scores for the TEL confidence scale, grade 8 technology and engineering literacy national assessment, by classification: 2014
ClassificationIndex valueTransformed scorePercentage of students
Can't-2.70874.6#
-2.42985.1#
-2.22315.6#
-2.06075.9#
-1.92576.1#
-1.80846.4#
-1.70316.6#
-1.60626.8#
-1.51497.01
Maybe-1.42777.11
-1.34307.31
-1.25997.51
-1.17757.61
-1.09517.82
-1.01238.02
-0.92858.12
-0.84348.34
-0.75658.53
-0.66778.74
-0.57678.84
-0.48309.05
-0.38629.25
-0.28579.46
-0.18079.66
Can-0.07019.98
0.047810.16
0.175210.46
0.315710.66
0.474610.96
0.660811.35
0.889811.84
1.192312.43
1.632613.36

# Rounds to zero.
NOTE: Detail may not sum to totals due to rounding. In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.


Index values and transformed scores for the technology and society in-school learning scale, grade 8 technology and engineering literacy national assessment, by classification: 2014
ClassificationIndex valueTransformed scorePercentage of students
Never-2.31525.41
-1.95476.11
-1.68336.61
Rarely-1.46237.11
-1.27087.52
-1.09677.82
-0.93268.14
-0.77338.54
-0.61458.87
Sometimes-0.45299.18
-0.28559.49
-0.10959.810
0.078010.211
0.279810.611
0.500011.010
Often0.745111.57
1.027012.16
1.369912.74
1.824813.62
NOTE: Detail may not sum to totals due to rounding. In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

Top

Index values and transformed scores for the technology and society out-of-school learning scale, grade 8 technology and engineering literacy national assessment, by classification: 2014
ClassificationIndex valueTransformed scorePercentage of students
Never-1.93826.18
-1.49117.03
Rarely-1.14257.74
-0.84428.34
-0.57188.912
-0.31189.48
Sometimes-0.05579.911
0.202810.410
0.469910.914
0.753011.57
Often1.064312.17
1.427212.94
1.889913.86

NOTE: Detail may not sum to totals due to rounding. In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.


Index values and transformed scores for the design and systems in-school learning scale, grade 8 technology and engineering literacy national assessment, by classification: 2014
ClassificationIndex valueTransformed scorePercentage of students
Never-1.96546.12
-1.53846.92
-1.21697.63
Rarely-0.95378.15
-0.72578.56
-0.52099.07
-0.33269.38
-0.15689.79
0.009410.010
Sometimes0.168510.39
0.322710.69
0.474810.97
0.627911.36
0.786211.65
0.955011.94
Often1.142812.33
1.363312.72
1.641213.31
2.026314.11

NOTE: Detail may not sum to totals due to rounding. In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.


Index values and transformed scores for the design and systems out-of-school learning scale, grade 8 technology and engineering literacy national assessment, by classification: 2014
ClassificationIndex valueTransformed scorePercentage of students
Never-1.93926.14
-1.51547.03
-1.20257.64
One to two times
-0.95218.15
-0.73968.56
-0.55168.98
-0.37939.27
-0.21659.68
Three to five times-0.05819.98
0.100710.28
0.265110.58
0.441710.96
0.640111.36
Greater than five times0.876711.85
1.182412.45
1.622813.29

NOTE: Detail may not sum to totals due to rounding. In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.


Index values and transformed scores for the computer use scale, grade 8 technology and engineering literacy national assessment, by classification: 2014
ClassificationIndex valueTransformed scorePercentage of students
Never-1.63566.73
-1.15287.74
-0.81268.47
Few times a year-0.55618.97
-0.35259.39
-0.18429.68
-0.04019.910
0.086710.27
0.201410.48
One to two times a month0.307310.67
0.407110.86
0.502911.05
0.596311.26
0.688911.43
0.782011.63
One to two times a week0.876811.82
0.974911.92
1.078012.21
1.188112.41
1.308312.61
1.442612.91
Daily1.597813.2#
1.784813.6#
2.024314.0#
2.354814.7#
# Rounds to zero.
NOTE: Detail may not sum to totals due to rounding. In 2014, grades 4 and 12 were not assessed.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

Top



Last updated 31 August 2021 (SK)