View source for SEBoK Review and Adjudication

The participation of the worldwide systems engineering community in the open collaboration process played a significant role in shaping the development versions of SEBoK leading to release of v. 1.0 in September 2012 (see [[Release History]]). It is expected that community participation and review of the content will continue to keep the content of the SEBoK useful, current, and relevant.

This article outlines the process that was followed to solicit and adjudicate review comments that have been received on versions 0.25, 0.5, 0.75, 1.0 and 1.0.1 of the SEBoK . A comprehensive list of all comments received and the authors’ response to those comments is also included in adjudication matrices. 

Since January 2013, versions 1.1 and beyond are being released under a new governance structure. The adjudication process described here is not applicable to for these releases. Along with SEBoK v. 1.1.2, the  [http://www.sebokwiki.org/sandbox SEBoK Sandbox] has been setup to receive comments and other contributions. 

==Reviewers and Review Comments==

===SEBoK v. 0.25===
The first version of the SEBoK, version 0.25, was released as a document for limited review in September 2010. A list of potential reviewers was compiled based on 
*direct requests to professional societies, corporations, and academic institutions; and
*volunteers who responded to requests during SEBoK related presentations and talks at conferences and other events.
The SEBoK v. 0.25 document, along with a review form with over thirty questions and detailed instructions were distributed to the potential reviewers. 

Reviews were received from 114 reviewers who provided over 3000 comments by December 2010.

===SEBoK v. 0.5===
A number of changes in the structure and content of the SEBoK was made in response to the review comments received for SEBoK v. 0.25. The next version of the SEBoK, version 0.5, was released in September 2011, as a wiki. In addition to providing easy access and navigation of the SEBoK, the wiki platform also enabled broader review by the worldwide systems engineering community. Release information and requests for review were sent out to an updated list of reviewers. Review comments on individual articles were predominantly received on the wiki. A review form was also distributed to receive overarching comments on the SEBoK as a whole and on the wiki implementation. Comments were also received in various other formats including emails, spreadsheets and marked-up documents until December 2011.

===SEBoK v. 0.75===
Though not originally planned, the third interim version of the SEBoK, version 0.75, was released in March 2012 for a short limited period review until April 2012.  Not all of the review comments were addressed in SEBoK v. 0.75. The wiki platform was well received by the worldwide SE community and so was retained for future releases of the SEBoK.  More features for reviewing and navigating were added to the SEBoK wiki. The wiki remained open for review, though it was stated that the adjudication of those comments may not be addressed for SEBoK v. 1.0.

===SEBoK v. 1.0 and v. 1.0.1===
The first production version of the SEBoK, version 1.0, was released in September 2012. In November 2012, version 1.0.1, a micro update was released to correct a few errors but with no significant change in the content. Unlike previous developmental versions of the SEBoK, public review of SEBoK v. 1.0 and v. 1.0.1 was not intentionally solicited but users were strongly encouraged to provide their comments and feedback.

===SEBoK v. 1.1, v. 1.1.1, and v. 1.1.2===
Beginning in January 2013, the SEBoK began operating under a new governance structure, led by INCOSE, IEEE-CS, and the SERC acting jointly as stewards (see [[Acknowledgements]]). Version 1.1 was the first SEBoK release under this new governance structure and the newly formed BKCASE Editorial Board that is still evolving. As a result, adjudication of some comments received on versions 1.0 and 1.0.1 is being deferred to future versions of the SEBoK.

The wiki continues to remain open to review comments on the latest release, SEBoK v. 1.1.2. Comments received will be adjudicated and reported in the next minor version of the SEBoK, version 1.2. With the publishing of SEBoK v. 1.1.2, contributions will also be possible through the [http://www.sebokwiki.org/sandbox SEBoK Sandbox].

==Adjudication Process==
A thorough process involving the authors and Part Team Leads (PTLs) was used to address every review comment received on the SEBoK. This process largely remained the same for all versions of the SEBoK that were released. Since January 2013, a new governance structure is in place where there are no PTLs. However, the overall process was retained for adjudicating comments received on versions 1.0 and 1.0.1. 

The following broad steps were followed for adjudication:
#All comments received in various formats were all compiled in matrices as spreadsheets, segregated by SEBoK parts. When warranted, compound comments were broken down into individual comments to support effective response to the comments. 
#The comments were then distributed to the respective PTLs and authors.
#The authors responded to every review comment by choosing one of the adjudication actions listed in Table 1 below. Where required, authors provided reasons and rationale for their decision on particular review comments.
#Adjudication matrices were then re-compiled incorporating author responses as spreadsheets for further use and distribution.


<center>
{|
|+'''Table 1. Types of Adjudication Actions.''' (SEBoK Original)
! Action 
! Description
|-
|'''Accepted'''
|
*The comment from the reviewer has been accepted in '''whole''', and suggested action has been taken.  
|-
| '''Modified'''
|
*The comment from the reviewer has been accepted in '''part''' and suggested action has been taken with some modification(s).
*Details of the modification is provided. 
|-
|'''Declined'''
|
*Suggestion recommended by the reviewer has been declined.
*Rationale for this decision is provided
|-
|''' Noted'''
|
*The comment made by the reviewer requires no further action. 
|-
| '''Copy'''
|
*This comment is similar to a comment already received and adjudicated (in this or earlier version of the SEBoK).  
|-
| '''Deferred'''
|
*Response to the comment is deferred to a future version of the SEBoK.  
|}
</center>

==Adjudication Matrices==

A detailed adjudication report was prepared based on the reviews received on SEBoK v. 0.25. For the review comments on the 0.5 and 0.75 versions of the SEBoK, the adjudication matrices have been prepared. Adjudication of comments received on the 1.0 and 1.0.1 versions are included in a single matrix. All matrices be may be downloaded (in PDF format) from the following links:

* [http://sebokwiki.org/downloads/SEBOKVersion0.25AdjudicationReportFINAL1.pdf SEBoK v. 0.25 Adjudication Report]
* [http://sebokwiki.org/downloads/SEBoK05_AdjudicationMatrix_FINAL.pdf SEBoK v. 0.5 Adjudication Matrix]
* [http://sebokwiki.org/downloads/SEBoK075_AdjudicationMatrix_FINAL.pdf SEBoK v. 0.75 Adjudication Matrix]
* [http://sebokwiki.org/downloads/SEBoK1.0_1.0.1_Adjudication%20Matrix.pdf SEBoK v. 1.0 & 1.0.1 Adjudication Matrix]

It may be noted that some review comments may be relevant only to the SEBoK version on which the comments were made; content may have changed since then. Names of individual reviewers have not been included in any of the above adjudication matrices. If you reviewed an earlier version of the SEBoK and would like to know your reviewer number and see the responses to your comments, please contact [mailto:bkcase@stevens.edu bkcase@stevens.edu].

{{DISQUS}}