Merge pull request #430 from serde-rs/bump

Update syntex to 0.38

README.md

@@ -12,74 +12,100 @@ information. In many situations, the handshake protocol between serializers and
 serializees can be completely optimized away, leaving Serde to perform roughly
 the same speed as a hand written serializer for a specific type.
 
-Documentation is available at:
+[Documentation](https://serde-rs.github.io/serde/serde/index.html)
 
-* [serde](https://serde-rs.github.io/serde/serde/index.html)
+Simple Serde Example
+====================
 
-Using Serde with Nightly Rust and serde\_macros
+Here is a simple example that uses
-===============================================
+[serde_json](https://github.com/serde-rs/json), which uses Serde under the
-
+covers, to generate and parse JSON. First, lets start off with the `Cargo.toml`
-Here is a simple example that demonstrates how to use Serde by serializing and
+file:
-deserializing to JSON. Serde comes with some powerful code generation libraries
-that work with Stable and Nightly Rust that eliminate much of the complexity of
-hand rolling serialization and deserialization for a given type. First lets see
-how we would use Nightly Rust, which is currently a bit simpler than Stable
-Rust:
-
-`Cargo.toml`:
 
 ```toml
 [package]
-name = "serde_example_nightly"
+name = "serde_example"
 version = "0.1.0"
 authors = ["Erick Tryzelaar <erick.tryzelaar@gmail.com>"]
 
 [dependencies]
-serde = "*"
 serde_json = "*"
-serde_macros = "*"
 ```
 
-`src/main.rs`
+Next, the `src/main.rs` file itself:
-
-```rust
-#![feature(custom_derive, plugin)]
-#![plugin(serde_macros)]
 
+```rust,ignore
 extern crate serde_json;
 
-#[derive(Serialize, Deserialize, Debug)]
+use std::collections::HashMap;
-struct Point {
+use serde_json::Value;
-    x: i32,
+use serde_json::builder::{ArrayBuilder, ObjectBuilder};
-    y: i32,
-}
 
 fn main() {
-    let point = Point { x: 1, y: 2 };
+    // Serde has support for many of the builtin Rust types, like arrays..:
-    let serialized = serde_json::to_string(&point).unwrap();
+    let v = vec![1, 2];
+    let serialized = serde_json::to_string(&v).unwrap();
+    println!("serialized vec: {:?}", serialized);
 
-    println!("{}", serialized);
+    let deserialized: Vec<u32> = serde_json::from_str(&serialized).unwrap();
+    println!("deserialized vec: {:?}", deserialized);
 
-    let deserialized: Point = serde_json::from_str(&serialized).unwrap();
+    // ... and maps:
+    let mut map = HashMap::new();
+    map.insert("x".to_string(), 1);
+    map.insert("y".to_string(), 2);
 
-    println!("{:?}", deserialized);
+    let serialized = serde_json::to_string(&map).unwrap();
+    println!("serialized map: {:?}", serialized);
+
+    let deserialized: HashMap<String, u32> = serde_json::from_str(&serialized).unwrap();
+    println!("deserialized map: {:?}", deserialized);
+
+    // It also can handle complex objects:
+    let value = ObjectBuilder::new()
+        .insert("int", 1)
+        .insert("string", "a string")
+        .insert("array", ArrayBuilder::new()
+                .push(1)
+                .push(2)
+                .unwrap())
+        .unwrap();
+
+    let serialized = serde_json::to_string(&value).unwrap();
+    println!("serialized value: {:?}", serialized);
+
+    let deserialized: serde_json::Value = serde_json::from_str(&serialized).unwrap();
+    println!("deserialized value: {:?}", deserialized);
 }
 ```
 
-When run, it produces:
+This produces the following output when run:
 
 ```
 % cargo run
-{"x":1,"y":2}
+serialized vec: "[1,2]"
-Point { x: 1, y: 2 }
+deserialized vec: [1, 2]
+serialized map: "{\"y\":2,\"x\":1}"
+deserialized map: {"y": 2, "x": 1}
+serialized value: "{\"array\":[1,2],\"int\":1,\"string\":\"a string\"}"
+deserialized value: {"array":[1,2],"int":1,"string":"a string"}
 ```
 
 Using Serde with Stable Rust and serde\_codegen
 ===============================================
 
-Stable Rust is a little more complicated because it does not yet support
+The example before used `serde_json::Value` as the in-memory representation of
-compiler plugins. Instead we need to use `serde_codegen` which is based on the
+the JSON value, but it's also possible for Serde to serialize to and from
-code generation library [syntex](https://github.com/serde-rs/syntex):
+regular Rust types. However, the code to do this can be a bit complicated to
+write. So instead, Serde also has some powerful code generation libraries that
+work with Stable and Nightly Rust that eliminate much of the complexity of hand
+rolling serialization and deserialization for a given type.
+
+First lets see how we would use Stable Rust, which is currently a tad more
+complicated than Nightly Rust due to having to work around compiler plugins
+being unstable.  We will use `serde_codegen` which is based on the code
+generation library [syntex](https://github.com/serde-rs/syntex). First we need
+to setup the `Cargo.toml` that builds the project:
 
 ```toml
 [package]
@@ -96,16 +122,8 @@ serde = "*"
 serde_json = "*"
 ```
 
-`src/main.rs`:
+Next, we define our source file, `src/main.rs.in`. Note this is a different
-
+extension than usual becaues we need to do code generation:
-```rust,ignore
-extern crate serde;
-extern crate serde_json;
-
-include!(concat!(env!("OUT_DIR"), "/main.rs"));
-```
-
-`src/main.rs.in`:
 
 ```rust,ignore
 #[derive(Serialize, Deserialize, Debug)]
@@ -116,17 +134,28 @@ struct Point {
 
 fn main() {
     let point = Point { x: 1, y: 2 };
-    let serialized = serde_json::to_string(&point).unwrap();
 
+    let serialized = serde_json::to_string(&point).unwrap();
     println!("{}", serialized);
 
     let deserialized: Point = serde_json::from_str(&serialized).unwrap();
-
     println!("{:?}", deserialized);
 }
 ```
 
-`build.rs`
+To finish up the main source code, we define a very simple `src/main.rs` that
+uses the generated code.
+
+`src/main.rs`:
+
+```rust,ignore
+extern crate serde;
+extern crate serde_json;
+
+include!(concat!(env!("OUT_DIR"), "/main.rs"));
+```
+
+The last step is to actually drive the code generation, with the `build.rs` script:
 
 ```rust,ignore
 extern crate serde_codegen;
@@ -144,7 +173,7 @@ pub fn main() {
 }
 ```
 
-This also produces:
+All this produces this when run:
 
 ```
 % cargo run
@@ -153,8 +182,63 @@ Point { x: 1, y: 2 }
 ```
 
 While this works well with Stable Rust, be aware that the error locations
-currently are reported in the generated file instead of in the source file. You
+currently are reported in the generated file instead of in the source file.
-may find it easier to develop with Nightly Rust and `serde\_macros`, then
+
+Using Serde with Nightly Rust and serde\_macros
+===============================================
+
+The prior example is a bit more complicated than it needs to be due to compiler
+plugins being unstable. However, if you are already using Nightly Rust, you can
+use `serde_macros`, which has a much simpler interface. First, here is the new
+`Cargo.toml`:
+
+```toml
+[package]
+name = "serde_example_nightly"
+version = "0.1.0"
+authors = ["Erick Tryzelaar <erick.tryzelaar@gmail.com>"]
+
+[dependencies]
+serde = "*"
+serde_json = "*"
+serde_macros = "*"
+```
+
+Note that it doesn't need a build script. Now the `src/main.rs`, which enables
+the plugin feature, and registers the `serde_macros` plugin:
+
+```rust
+#![feature(custom_derive, plugin)]
+#![plugin(serde_macros)]
+
+extern crate serde_json;
+
+#[derive(Serialize, Deserialize, Debug)]
+struct Point {
+    x: i32,
+    y: i32,
+}
+
+fn main() {
+    let point = Point { x: 1, y: 2 };
+
+    let serialized = serde_json::to_string(&point).unwrap();
+    println!("{}", serialized);
+
+    let deserialized: Point = serde_json::from_str(&serialized).unwrap();
+    println!("{:?}", deserialized);
+}
+```
+
+This also produces the same output:
+
+```
+% cargo run
+{"x":1,"y":2}
+Point { x: 1, y: 2 }
+```
+
+You may find it easier to develop with Nightly Rust and `serde\_macros`, then
 deploy with Stable Rust and `serde_codegen`. It's possible to combine both
 approaches in one setup:
 

examples/serde-syntex-example/Cargo.toml

@@ -9,9 +9,9 @@ default = ["serde_codegen"]
 nightly = ["serde_macros"]
 
 [build-dependencies]
-serde_codegen = { version = "^0.7.12", optional = true }
+serde_codegen = { version = "^0.7", optional = true }
 
 [dependencies]
-serde = "^0.7.12"
+serde = "^0.7"
-serde_json = "^0.7.0"
+serde_json = "^0.7"
-serde_macros = { version = "^0.7.12", optional = true }
+serde_macros = { version = "^0.7", optional = true }

serde/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "serde"
-version = "0.7.12"
+version = "0.7.14"
 authors = ["Erick Tryzelaar <erick.tryzelaar@gmail.com>"]
 license = "MIT/Apache-2.0"
 description = "A generic serialization/deserialization framework"

serde/src/de/mod.rs

@@ -8,6 +8,8 @@ use error;
 #[cfg(all(not(feature = "std"), feature = "collections"))]
 use collections::{String, Vec};
 
+use core::fmt;
+
 pub mod impls;
 pub mod value;
 mod from_primitive;
@@ -169,6 +171,46 @@ pub enum Type {
     Bytes,
 }
 
+impl fmt::Display for Type {
+    fn fmt(&self, formatter: &mut fmt::Formatter) -> Result<(), fmt::Error> {
+        let display = match *self {
+            Type::Bool          => "bool",
+            Type::Usize         => "usize",
+            Type::U8            => "u8",
+            Type::U16           => "u16",
+            Type::U32           => "u32",
+            Type::U64           => "u64",
+            Type::Isize         => "isize",
+            Type::I8            => "i8",
+            Type::I16           => "i16",
+            Type::I32           => "i32",
+            Type::I64           => "i64",
+            Type::F32           => "f32",
+            Type::F64           => "f64",
+            Type::Char          => "char",
+            Type::Str           => "str",
+            Type::String        => "string",
+            Type::Unit          => "unit",
+            Type::Option        => "option",
+            Type::Seq           => "seq",
+            Type::Map           => "map",
+            Type::UnitStruct    => "unit struct",
+            Type::NewtypeStruct => "newtype struct",
+            Type::TupleStruct   => "tuple struct",
+            Type::Struct        => "struct",
+            Type::FieldName     => "field name",
+            Type::Tuple         => "tuple",
+            Type::Enum          => "enum",
+            Type::VariantName   => "variant name",
+            Type::StructVariant => "struct variant",
+            Type::TupleVariant  => "tuple variant",
+            Type::UnitVariant   => "unit variant",
+            Type::Bytes         => "bytes",
+        };
+        display.fmt(formatter)
+    }
+}
+
 ///////////////////////////////////////////////////////////////////////////////
 
 /// `Deserialize` represents a type that can be deserialized.

serde_codegen/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "serde_codegen"
-version = "0.7.12"
+version = "0.7.14"
 authors = ["Erick Tryzelaar <erick.tryzelaar@gmail.com>"]
 license = "MIT/Apache-2.0"
 description = "Macros to auto-generate implementations for the serde framework"
@@ -24,14 +24,14 @@ with-syntex = [
 ]
 
 [build-dependencies]
-quasi_codegen = { version = "^0.14.0", optional = true }
+quasi_codegen = { version = "^0.15.0", optional = true }
-syntex = { version = "^0.37.0", optional = true }
+syntex = { version = "^0.38.0", optional = true }
 
 [dependencies]
-aster = { version = "^0.20.0", default-features = false }
+aster = { version = "^0.21.1", default-features = false }
 clippy = { version = "^0.*", optional = true }
-quasi = { version = "^0.14.0", default-features = false }
+quasi = { version = "^0.15.0", default-features = false }
-quasi_macros = { version = "^0.14.0", optional = true }
+quasi_macros = { version = "^0.15.0", optional = true }
-serde_codegen_internals = { version = "^0.2.0", path = "../serde_codegen_internals", default-features = false }
+serde_codegen_internals = { version = "^0.3.0", path = "../serde_codegen_internals", default-features = false }
-syntex = { version = "^0.37.0", optional = true }
+syntex = { version = "^0.38.0", optional = true }
-syntex_syntax = { version = "^0.37.0", optional = true }
+syntex_syntax = { version = "^0.38.0", optional = true }

serde_codegen/src/de.rs

@@ -8,6 +8,7 @@ use syntax::ptr::P;
 use syntax::tokenstream::TokenTree;
 
 use bound;
+use span;
 use internals::ast::{Body, Field, Item, Style, Variant};
 use internals::{attr, Error};
 
@@ -44,7 +45,7 @@ pub fn expand_derive_deserialize(
     let builder = aster::AstBuilder::new().span(span);
 
     let impl_item = deserialize_item(cx, &builder, &item);
-    push(Annotatable::Item(impl_item))
+    push(span::record_expansion(cx, impl_item, "Deserialize"))
 }
 
 fn deserialize_item(
@@ -394,6 +395,7 @@ fn deserialize_seq(
     fields: &[Field],
     is_struct: bool,
 ) -> P<ast::Expr> {
+    let mut index_in_seq = 0usize;
     let let_values: Vec<_> = fields.iter()
         .enumerate()
         .map(|(i, field)| {
@@ -419,14 +421,17 @@ fn deserialize_seq(
                         })
                     }
                 };
-                quote_stmt!(cx,
+                let assign = quote_stmt!(cx,
                     let $name = match $visit {
                         Some(value) => { value },
                         None => {
-                            return Err(_serde::de::Error::end_of_stream());
+                            try!(visitor.end());
+                            return Err(_serde::de::Error::invalid_length($index_in_seq));
                         }
                     };
-                ).unwrap()
+                ).unwrap();
+                index_in_seq += 1;
+                assign
             }
         })
         .collect();
@@ -1092,10 +1097,10 @@ fn deserialize_map(
             }
         }
 
-        $extract_values
-
         try!(visitor.end());
 
+        $extract_values
+
         Ok($result)
     })
 }

serde_codegen/src/lib.rs.in

@@ -1,3 +1,4 @@
 mod bound;
 mod de;
 mod ser;
+mod span;

serde_codegen/src/ser.rs

@@ -6,6 +6,7 @@ use syntax::ext::base::{Annotatable, ExtCtxt};
 use syntax::ptr::P;
 
 use bound;
+use span;
 use internals::ast::{Body, Field, Item, Style, Variant};
 use internals::{attr, Error};
 
@@ -38,7 +39,7 @@ pub fn expand_derive_serialize(
     let builder = aster::AstBuilder::new().span(span);
 
     let impl_item = serialize_item(cx, &builder, &item);
-    push(Annotatable::Item(impl_item))
+    push(span::record_expansion(cx, impl_item, "Serialize"))
 }
 
 fn serialize_item(

serde_codegen/src/span.rs

@@ -0,0 +1,43 @@
+use syntax::ast;
+use syntax::codemap::{self, ExpnId, Span};
+use syntax::ext::base::{Annotatable, ExtCtxt};
+use syntax::fold::{self, Folder};
+use syntax::parse::token::intern;
+use syntax::ptr::P;
+
+pub fn record_expansion(
+    cx: &ExtCtxt,
+    item: P<ast::Item>,
+    derive: &str,
+) -> Annotatable {
+    let info = codemap::ExpnInfo {
+        call_site: codemap::DUMMY_SP,
+        callee: codemap::NameAndSpan {
+            format: codemap::MacroAttribute(intern(&format!("derive({})", derive))),
+            span: None,
+            allow_internal_unstable: false,
+        },
+    };
+    let expn_id = cx.codemap().record_expansion(info);
+
+    let mut respanner = Respanner { expn_id: expn_id };
+    let item = item.map(|item| respanner.fold_item_simple(item));
+    Annotatable::Item(item)
+}
+
+struct Respanner {
+    expn_id: ExpnId,
+}
+
+impl Folder for Respanner {
+    fn new_span(&mut self, span: Span) -> Span {
+        Span {
+            expn_id: self.expn_id,
+            .. span
+        }
+    }
+
+    fn fold_mac(&mut self, mac: ast::Mac) -> ast::Mac {
+        fold::noop_fold_mac(mac, self)
+    }
+}

serde_codegen_internals/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "serde_codegen_internals"
-version = "0.2.0"
+version = "0.3.0"
 authors = ["Erick Tryzelaar <erick.tryzelaar@gmail.com>"]
 license = "MIT/Apache-2.0"
 description = "AST representation used by Serde codegen. Unstable."
@@ -16,5 +16,5 @@ with-syntex = ["syntex_syntax", "syntex_errors"]
 
 [dependencies]
 clippy = { version = "^0.*", optional = true }
-syntex_syntax = { version = "^0.37.0", optional = true }
+syntex_syntax = { version = "^0.38.0", optional = true }
-syntex_errors = { version = "^0.37.0", optional = true }
+syntex_errors = { version = "^0.38.0", optional = true }

serde_codegen_internals/src/attr.rs

@@ -1,3 +1,5 @@
+use std::rc::Rc;
+
 use syntax::ast;
 use syntax::attr::{self, HasAttrs};
 use syntax::codemap::{Span, Spanned, respan};
@@ -517,33 +519,33 @@ struct Respanner<'a, 'b: 'a> {
 }
 
 impl<'a, 'b> Folder for Respanner<'a, 'b> {
-    fn fold_tt(&mut self, tt: TokenTree) -> TokenTree {
+    fn fold_tt(&mut self, tt: &TokenTree) -> TokenTree {
-        match tt {
+        match *tt {
             TokenTree::Token(span, ref tok) => {
                 TokenTree::Token(
                     self.new_span(span),
                     self.fold_token(tok.clone())
                 )
             }
-            TokenTree::Delimited(span, delimed) => {
+            TokenTree::Delimited(span, ref delimed) => {
                 TokenTree::Delimited(
                     self.new_span(span),
-                    tokenstream::Delimited {
+                    Rc::new(tokenstream::Delimited {
                         delim: delimed.delim,
                         open_span: delimed.open_span,
-                        tts: self.fold_tts(delimed.tts),
+                        tts: self.fold_tts(&delimed.tts),
                         close_span: delimed.close_span,
-                    }
+                    })
                 )
             }
-            TokenTree::Sequence(span, seq) => {
+            TokenTree::Sequence(span, ref seq) => {
                 TokenTree::Sequence(
                     self.new_span(span),
-                    tokenstream::SequenceRepetition {
+                    Rc::new(tokenstream::SequenceRepetition {
-                        tts: self.fold_tts(seq.tts),
+                        tts: self.fold_tts(&seq.tts),
                         separator: seq.separator.clone().map(|tok| self.fold_token(tok)),
-                        ..seq
+                        ..**seq
-                    }
+                    })
                 )
             }
         }
@@ -589,7 +591,7 @@ fn parse_string_via_tts<T, F>(cx: &ExtCtxt, name: &str, string: String, action:
         cx.parse_sess()));
 
     // Respan the spans to say they are all coming from this macro.
-    let tts = Respanner { cx: cx }.fold_tts(tts);
+    let tts = Respanner { cx: cx }.fold_tts(&tts);
 
     let mut parser = parse::new_parser_from_tts(cx.parse_sess(), cx.cfg(), tts);
 

serde_macros/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "serde_macros"
-version = "0.7.12"
+version = "0.7.14"
 authors = ["Erick Tryzelaar <erick.tryzelaar@gmail.com>"]
 license = "MIT/Apache-2.0"
 description = "Macros to auto-generate implementations for the serde framework"
@@ -18,14 +18,15 @@ nightly-testing = ["clippy", "serde/nightly-testing", "serde_codegen/nightly-tes
 
 [dependencies]
 clippy = { version = "^0.*", optional = true }
-serde_codegen = { version = "^0.7.12", path = "../serde_codegen", default-features = false, features = ["nightly"] }
+serde_codegen = { version = "^0.7.14", path = "../serde_codegen", default-features = false, features = ["nightly"] }
 
 [dev-dependencies]
+clippy = "^0.0.78"
 compiletest_rs = "^0.2.0"
 fnv = "1.0"
 rustc-serialize = "^0.3.16"
-serde = { version = "^0.7.12", path = "../serde" }
+serde = { version = "^0.7.14", path = "../serde" }
-serde_test = { version = "^0.7.12", path = "../serde_test" }
+serde_test = { version = "^0.7.14", path = "../serde_test" }
 
 [[test]]
 name = "test"

serde_macros/tests/compile_tests.rs

@@ -20,6 +20,11 @@ fn run_mode(mode: &'static str) {
 }
 
 #[test]
-fn compile_test() {
+fn compile_fail() {
     run_mode("compile-fail");
 }
+
+#[test]
+fn run_pass() {
+    run_mode("run-pass");
+}

serde_macros/tests/run-pass/identity-op.rs

@@ -0,0 +1,12 @@
+#![feature(custom_derive, plugin)]
+#![plugin(serde_macros, clippy)]
+
+#![deny(identity_op)]
+
+// The derived implementation uses 0+1 to add up the number of fields
+// serialized, which Clippy warns about. If the expansion info is registered
+// correctly, the Clippy lint is not triggered.
+#[derive(Serialize)]
+struct A { b: u8 }
+
+fn main() {}

serde_test/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "serde_test"
-version = "0.7.12"
+version = "0.7.14"
 authors = ["Erick Tryzelaar <erick.tryzelaar@gmail.com>"]
 license = "MIT/Apache-2.0"
 description = "Token De/Serializer for testing De/Serialize implementations"
@@ -11,4 +11,4 @@ keywords = ["serde", "serialization"]
 include = ["Cargo.toml", "src/**/*.rs"]
 
 [dependencies]
-serde = { version = "0.7.12", path = "../serde" }
+serde = { version = "0.7.14", path = "../serde" }

testing/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "serde_testing"
-version = "0.7.12"
+version = "0.7.14"
 authors = ["Erick Tryzelaar <erick.tryzelaar@gmail.com>"]
 license = "MIT/Apache-2.0"
 description = "A generic serialization/deserialization framework"

testing/tests/test_annotations.rs

@@ -946,3 +946,31 @@ fn test_missing_renamed_field_enum() {
         Error::MissingField("d"),
     );
 }
+
+#[derive(Debug, PartialEq, Deserialize)]
+enum InvalidLengthEnum {
+    A(i32, i32, i32),
+    B(#[serde(skip_deserializing)] i32, i32, i32),
+}
+
+#[test]
+fn test_invalid_length_enum() {
+    assert_de_tokens_error::<InvalidLengthEnum>(
+        &[
+            Token::EnumSeqStart("InvalidLengthEnum", "A", Some(3)),
+                Token::EnumSeqSep,
+                Token::I32(1),
+            Token::EnumSeqEnd,
+        ],
+        Error::InvalidLength(1),
+    );
+    assert_de_tokens_error::<InvalidLengthEnum>(
+        &[
+            Token::EnumSeqStart("InvalidLengthEnum", "B", Some(3)),
+                Token::EnumSeqSep,
+                Token::I32(1),
+            Token::EnumSeqEnd,
+        ],
+        Error::InvalidLength(1),
+    );
+}