Thursday, December 22, 2016

Technical Note TN2265: Troubleshooting Push Notifications

Technical Note TN2265: Troubleshooting Push Notifications: Technical Note TN2265
Troubleshooting Push Notifications

微信协议简单调研笔记 - 聂永的博客 - BlogJava

微信协议简单调研笔记 - 聂永的博客 - BlogJava: 微信协议简单调研笔记
聂永的博客
记录工作/学习的点点滴滴。
微信协议简单调研笔记

前言

微信可调研点很多,这里仅仅从协议角度进行调研,会涉及到微信协议交换、消息收发等。所谓“弱水三千,只取一瓢”吧。

杂七杂八的,有些长,可直接拉到最后看结论好了。

一。微信协议概览

微信传输协议,官方公布甚少,在微信技术总监所透漏PPT《微信之道—至简》文档中,有所体现。

纯个人理解:

因张小龙做邮箱Foxmail起家,继而又做了QQ Mail等,QQ Mail是国内第一个支持Exchange ActiveSync协议的免费邮箱,基于其从业背景,微信从一开始就采取基于ActiveSync的修改版状态同步协议Sync,也就再自然不过了。
一句话:增量式、按序、可靠的状态同步传输的微信协议。

大致交换简图如下:

Image(9)

如何获取新数据呢:

服务器端通知,客户端获取
客户端携带最新的SyncKey,发起数据请求
服务器端生成最新的SyncKey连同最新数据发送给客户端
基于版本号机制同步协议,可确保数据增量、有序传输
SyncKey,由服务器端序列号生成器生成,一旦有新消息产生,将会产生最新的SyncKey。类似于版本号
服务器端通知有状态更新,客户端主动获取自从上次更新之后有变动的状态数据,增量式,顺序式。

二。微信Web端简单调试

在线版本微信:

https://webpush.weixin.qq.com/

通过Firefox + Firebug组合调试,也能证实了微信大致通过交换SyncKey方式获取新数据的论述。

1. 发起GET长连接检测是否存在新的需要同步的数据

会携带上最新SyncKey

https://webpush.weixin.qq.com/cgi-bin/mmwebwx-bin/synccheck?callback=jQuery18306073923335455973_1393208247730&r=1393209241862&sid=s7c%2FsxpGRSihgZAA&uin=937355&deviceid=e542565508353877&synckey=1_620943725%7C2_620943769%7C3_620943770%7C11_620942796%7C201_1393208420%7C202_1393209127%7C1000_1393203219&_=1393209241865
返回内容:

window.synccheck={retcode:"0",selector:"2"}
selector值大于0,表示有新的消息需要同步。

据目测,心跳周期为27秒左右。

2. 一旦有新数据,客户端POST请求主动获取同步的数据

https://webpush.weixin.qq.com/cgi-bin/mmwebwx-bin/webwxsync?sid=s7c%2FsxpGRSihgZAA&r=1393208447375
携带消息体:

{"BaseRequest":{"Uin":937355,"Sid":"s7c/sxpGRSihgZAA"},"SyncKey":{"Count":6,"List":[{"Key":1,"Val":620943725},{"Key":2,"Val":620943767},{"Key":3,"Val":620943760},{"Key":11,"Val":620942796},{"Key":201,"Val":1393208365},{"Key":1000,"Val":1393203219}]},"rr":1393208447374}
会携带上最新的SyncKey,会返回复杂结构体JSON内容。

但浏览端收取到消息之后,如何通知服务器端已确认收到了?Web版本微信,没有去做。

在以往使用过程中,曾发现WEB端有丢失消息的现象,但属于偶尔现象。但Android微信客户端(只要登陆连接上来之后)貌似就没有丢失过。

3. 发送消息流程

发起一个POST提交,用于提交用户需要发送的消息

https://webpush.weixin.qq.com/cgi-bin/mmwebwx-bin/webwxsendmsg?sid=lQ95vHR52DiaLVqo&r=1393988414386

发送内容:

{"BaseRequest":{"Uin":937355,"Sid":"lQ95vHR52DiaLVqo","Skey":"A6A1ECC6A7DE59DEFF6A05F226AA334DECBA457887B25BC6","DeviceID":"e937227863752975"},"Msg":{"FromUserName":"yongboy","ToUserName":"hehe057854","Type":1,"Content":"hello","ClientMsgId":1393988414380,"LocalID":1393988414380}}
相应内容:

{
"BaseResponse": {
"Ret": 0,
"ErrMsg": ""
}
,
"MsgID": 1020944348,
"LocalID": "1393988414380"
}
再次发起一个POST请求,用于申请最新SyncKey

https://webpush.weixin.qq.com/cgi-bin/mmwebwx-bin/webwxsync?sid=lQ95vHR52DiaLVqo&r=1393988414756

发送内容:

{"BaseRequest":{"Uin":937355,"Sid":"lQ95vHR52DiaLVqo"},"SyncKey":{"Count":6,"List":[{"Key":1,"Val":620944310},{"Key":2,"Val":620944346},{"Key":3,"Val":620944344},{"Key":11,"Val":620942796},{"Key":201,"Val":1393988357},{"Key":1000,"Val":1393930108}]},"rr":1393988414756}
响应的(部分)内容:

"SKey": "8F8C6A03489E85E9FDF727ACB95C93C2CDCE9FB9532FC15B"
终止GET长连接,使用最新SyncKey再次发起一个新的GET长连接

https://webpush.weixin.qq.com/cgi-bin/mmwebwx-bin/synccheck?callback=jQuery1830245810089652082181393988305564&r=1393988415015&sid=lQ95vHR52DiaLVqo&uin=937355&deviceid=e937227863752975&synckey=1620944310%7C2620944348%7C3620944344%7C11620942796%7C2011393988357%7C10001393930108&=1393988415016

三。微信Android简单分析

Windows桌面端Android虚拟机中运行最新版微信(5.2),通过tcpdump/Wireshark组合封包分析,以下为分析结果。

0. 初始连接记录

简单记录微信启动之后请求:

11:20:35 dns查询
dns.weixin.qq.com
返回一组IP地址

11:20:35 DNS查询
long.weixin.qq.com
返回一组IP地址,本次通信中,微信使用了最后一个IP作为TCP长连接的连接地址。

11:20:35
http://dns.weixin.qq.com/cgi-bin/micromsg-bin/newgetdns?uin=0&clientversion=620888113&scene=0&net=1
用于请求服务器获得最优IP路径。服务器通过结算返回一个xml定义了域名:IP对应列表。仔细阅读,可看到微信已经开始了国际化的步伐:香港、加拿大、韩国等。
具体文本,请参考:https://gist.github.com/yongboy/9341884

11:20:35
获取到long.weixin.qq.com最优IP,然后建立到101.227.131.105的TCP长连接

11:21:25
POST http://short.weixin.qq.com/cgi-bin/micromsg-bin/getprofile HTTP/1.1 (application/octet-stream)
返回一个名为“micromsgresp.dat”的附件,估计是未阅读的离线消息

11:21:31
POST http://short.weixin.qq.com/cgi-bin/micromsg-bin/whatsnews HTTP/1.1 (application/octet-stream)
大概是资讯、订阅更新等

中间进行一些资源请求等,类似于
GET http://wx.qlogo.cn/mmhead/Q3auHgzwzM7NR4TYFcoNjbxZpfO9aiaE7RU5lXGUw13SMicL6iacWIf2A/96
图片等一些静态资源都会被分配到wx.qlogo.cn域名下面

不明白做什么用途
POST http://short.weixin.qq.com/cgi-bin/micromsg-bin/downloadpackage HTTP/1.1 (application/octet-stream)
输出为micromsgresp.dat文件

11:21:47
GET http://support.weixin.qq.com/cgi-bin/mmsupport-bin/reportdevice?channel=34&deviceid=A952001f7a840c2a&clientversion=620888113&platform=0&lang=zh_CN&installtype=0 HTTP/1.1
返回chunked分块数据

11:21:49
POST http://short.weixin.qq.com/cgi-bin/micromsg-bin/reportstrategy HTTP/1.1 (application/octet-stream)
1. 心跳频率约为5分钟

上次使用Wireshark分析有误(得出18分钟结论),再次重新分析,心跳频率在5分钟左右。

2. 登陆之后,会建立一个长连接,端口号为8080

简单目测为HTTP,初始以为是双通道HTTP,难道是自定义的用于双通道通信的HTTP协议吗,网络上可见资料都是模棱两可、语焉不详。

具体查看长连接初始数据通信,没有发现任何包含"HTTP"字样的数据,以为是微信自定义的TCP/HTTP通信格式。据分析,用于可能用于获取数据、心跳交换消息等用途吧。这个后面会详谈微信是如何做到的。

2.0 初始消息传输
个人资料、离线未阅读消息部分等通过 POST HTTP短连接单独获取。

2.1 二进制简单分析
抽取微信某次HTTP协议方式通信数据,16进制表示,每两个靠近的数字为一个byte字节:

2014-03-03_15h07_30

微信协议可能如下:

一个消息包 = 消息头 + 消息体
消息头固定16字节长度,消息包长度定义在消息头前4个字节中。

单纯摘取第0000行为例,共16个字节的头部:

00 00 00 10 00 10 00 01 00 00 00 06 00 00 00 0f
16进制表示,每两个紧挨着数字代表一个byte字节。

微信消息包格式: 1. 前4字节表示数据包长度,可变 值为16时,意味着一个仅仅包含头部的完整的数据包(可能表示着预先定义好的业务意义),后面可能还有会别的消息包 2. 2个字节表示头部长度,固定值,0x10 = 16 3. 2个字节表示谢意版本,固定值,0x01 = 1 4. 4个字节操作说明数字,可变 5. 序列号,可变 6. 头部后面紧跟着消息体,非明文,加密形式 7. 一个消息包,最小16 byte字节

通过上图(以及其它数据多次采样)分析:

0000 - 0040为单独的数据包
0050行为下一个数据包的头部,前四个字节值为0xca = 202,表示包含了从0050-0110共202个字节数据
一次数据发送,可能包含若干子数据包
换行符\n,16进制表示为0x0a,在00f0行,包含了两个换行符号
一个数据体换行符号用于更细粒度的业务数据分割 是否蒙对,需要问问做微信协议的同学
所有被标记为HTTP协议通信所发送数据都包含换行符号
2.2 动手试试猜想,模拟微信TCP长连接
开始很不解为什么会出现如此怪异的HTTP双通道长连接请求,难道基于TCP通信,然后做了一些手脚?很常规的TCP长连接,传输数据时(不是所有数据传输),被wireshark误认为HTTP长连接。这个需要做一个实验证实一下自己想法,设想如下:

写一个Ping-Pong客户端、服务器端程序,然后使用Wireshark看一下结果,是否符合判断。

Java版本的请求端,默认请求8080端口:

好用到令人发指的 Surge 工具 - 简书

href="http://www.jianshu.com/p/6cc10eeef31a"
好用到令人发指的 Surge 工具
字数2656 阅读88988 评论24 喜欢150
Surge 这个是 iOS9 上的神器,作者虽然在官网说明 Surge 是开发者的网络调试工具,但是这个工具自打上架以来,其最广泛的应用场景绝对包括 iOS9梯子 这一刚需,iOS9 上有了 Surge 之后,iPhone 越狱的理由又少了一条。

写在最前面
如果只是想使用 Surge 来作 Shadowsocks 的梯子,而不想了解它的具体工作方式与配置,只需要四步操作:

iTunes上购买Surge软件。
载我整理好的 Surge 配置文件。(可以在 Surge 软件中直接 Download 或是用 iTunes 导入)
根据你的 SS 服务器信息,在Surge中修改配置。
选中配置后点 start 启动后按其提示操作。
操作完成之后,状态栏出现vpn字样,此时 iOS 设备即可以科学上网了,同时还自带各种视频客户端的广告过滤(媳妇刚需,码农泡妹必备)。

对 Surge 本身没有兴趣的同学,以下的内容可以直接忽略了。

主要功能
Surge 的主要功能包括以下:

截获 iOS 上的所有应用程序的 HTTP/HTTPS/TCP 流量,将其重定向到 HTTP/HTTPS/SOCK5 代理服务器转发。
重载 iOS 蜂窝网络下的 DNS 服务器配置,使得 iOS 设备工作在蜂窝网络之下时,开发者也能够配置 DNS 服务器。
记录设备所发出/接收的 HTTP 请求/应答首部信息。
配置基于 domain, domain suffix, domain keyword, CIDR IP 以及 GeoIP 的过滤规则。
对设备上 WIFI、蜂窝网络以及代理连接会话的流量进行统计以及测速。
从 URL 或 iTunes 导入/导出配置文件。
重度使用场景下有很高的性能以及适应性。
基于 domain 规则屏蔽广告。
完全兼容蜂窝网络。
与 iOS9之前需要越狱才能使用的 Shadowsocks 比较,Surge 在 iPhone 有强大的可配置性(过滤规则)、数据统计以及日志,较为遗憾的是由于 iOS 沙盘的限制,Surge 无法实现 应用内代理 功能,希望后续有对应的越狱插件可以弥补这个不足。

软件构架
为了让使用者能够更好地理解 Surge 的各项配置的作用,作者在官网上特意花了一个章节描述其软件构架


Surge Architecture
Surge 主要包括 Surge proxy server 与 Surge TUN interface 这两个组件,分别负责 HTTP/HTTPS 代理以及 IP 代理。

Surge Proxy server: Surge 功能开启后将自动代理 iOS 设备上所有的 HTTP/HTTPS ,同时对所有的 HTTP/HTTPS 代理使用同一个代理会话,以最高限度提高 Surge 的代理性能。Surge 可以通过 skip-proxy 指定哪些流量不被 proxy server 所代理

Surge TUN interface: iOS 上大部分应用程序的网络交互使用 HTTP/HTTPS,但也存在部分应用程序(如:iOS邮件客户端、Facebook客户端)等应用程序使用其他的通讯方式(如:SPDY等),这些应用无法被 Proxy server 所代理,这就需要使用更为底层的TUN interface 隧道方式。Surge 可以通过 bypass-tun 指定哪些数据流量不送 TUN interface 处理

注意: 当前 Surge TUN interface 只能处理 TCP 流量,对于 ICMP、UDP 流量将被直接丢弃,因此需要通过配置 bypass-tun 选项来放行这些流量。
配置文件
Surge 可以通过 URL下载 和 iTunes更新 这两种方式导入配置文件,配置文件由多个段(Section)构成,主要包括:General, Proxy, Rule等几个部分。

通用配置
通用配置段[general]的主要配置如下:

[General]
loglevel = verbose
bypass-system = true
skip-proxy = 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, localhost, *.local
bypass-tun = 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12
dns-server = 8.8.8.8, 8.8.4.4
loglevel
Surge输出的日志信息级别,按日志输出从多到少分别为 verbose,info,notify与warning,默认为 notify,一般情况下不要设置为 verbose ,这会大幅度降低 Surge 的性能,如果只是做为梯子使用,建议设置为输出最少的 warning 级别。

bypass-system
全局忽略配置,此配置列表中指定的服务器请求流量将不会被 Surge 所处理(包括 Proxy server 与 TUN interface),在一般使用中,请将以下服务器列入 bypass-system ,否则可能会出现一些奇怪问题(比如:应用程序的推送被延迟),添加 Apple 常用的服务器域名清单:

api.smoot.apple.com
configuration.apple.com
xp.apple.com
smp-device-content.apple.com
guzzoni.apple.com
captive.apple.com
*.ess.apple.com
*.push.apple.com
*.push-apple.com.akadns.net
同时也将 Apple 的服务器 IP 地址段过滤规则加上 IP-CIDR, 17.0.0.0/8, DIRECT, no-resolve 。

skip-proxy
skip-proxy 用于指定哪些服务器的域名和IP不被 Proxy server 代理,常用的有以下配置方式:

顶级域名,如 apple.com
包含特定关键字的域名,如 *apple.com
特定子域名,如 store.apple.com
特定的IP地址以及IP地址段,如 192.168.2.11,192.168.2.*,192.168.2.0/24
做梯子使用时,可以考虑在这里添加常用的应用的域名以及整个中国区域的 IP 地址段。

需要特别说明的是主流应用程序,特别是现象级的应用,一般都广泛采用 CDN 技术对服务器请求进行加速,这种情况下同个域名在不同的地区及同地区不同时间都可能会请求到不同 IP 地址,因此要通过添加域名的方式匹配,而不能采用简单地添加指定 IP 地址。
bypass-tun
bypass-tun 用于指定哪些服务器的域名/IP不被 TUN interface 转发,配置的方式同 skip-proxy,需要特别注意的是 Proxy server 所代理的流量不会被 TUNN interface 代理,因此在配置 bypass-tun 时需要结合 skip-proxy 进行综合考虑。

dns-server
dns-server 用于指定 iOS 设备所使用的域名解析服务器,iOS 本身是不允许修改蜂窝网络下的域名解析服务器,通过使用 Surge 的 dns-server 配置则可以达到修改蜂窝网络域名解析服务器的目的。

代理配置
代理配置段 [Proxy] 用于配置 HTTP, HTTPS 以及 SOCKS5 代理服务器(梯子的关键),配置的例子如下:

[Proxy]
ProxyA = socks5, example.server.com, 3129
ProxyB = http, example.server.com, 3128
ProxyC = https, example.server.com, 443, username, password
这个例子中,配置了三个服务器,分别对应于 HTTP, HTTPS 以及 SOCKS5 这三种代理机制,Surge 可以针对这三个代理指定多套不同的配置:

指定 HTTP/HTTPS 代理的用户名与密码(可选)
指定 HTTPS 代理的加密机制,如 ProxyC = https, example.server.com, 443, username, password, cipher=TLS_RSA_WITH_AES_128_GCM_SHA256
指定 TLS 会话重传,HTTPS 代理默认打开 False Start与ECN功能。(未来版本将支持 on/off 配置)
过滤规则
[Rule]段用来配置 Surge 的过滤规则,规则按序存储,匹配时从上到下,先匹配先生效,因此对于常用的规则应该放在前头,以提高性能,配置的例子如下:

[Rule]
DOMAIN-SUFFIX,company.com,ProxyA
DOMAIN-KEYWORD,google,DIRECT
GEOIP,US,DIRECT
IP-CIDR,192.168.0.0/16,DIRECT
FINAL ProxyB
Surge 支持 DOMAIN,DOMAIN-SUFFIX,DOMAIN-KEYWORD,GEOIP,IP-CIDR与FINAL 这六种规则,每条规则都由 类型,匹配域以及行为 这三个部分构成(除了 FINAL 类型外),每个部分使用逗号隔开,每条规则可指定的行为包括:代理(Proxy),不代理(DIRECT)和丢弃(REJECT)这三种。

六种过滤规则的配置如下:

DOMAIN,精确指定域名,DOMAIN,www.apple.com,Proxy,匹配所有发往 www.apple.com 的流量
DOMAIN-SUFFIX,按域名后缀匹配,DOMAIN-SUFFIX,apple.com,Proxy,匹配所有发往以 apple.com 结尾的流量,如 store.apple.com,mail.apple.com等,但不包括 issue-apple.com
DOMAIN-KEWORD,按域名关键字匹配,DOMAIN-KEYWORD,google,Proxy,匹配所有域名中包含 google 的流量,如: www.google.com, issue-google.cn 等
IP-CIDR,按 IP 地址无类匹配,IP-CIDR, 192.168.0.0/16,DIRECT,匹配所有到内网 192.168.0.0/16 的流量
GEOIP,按地理位置IP匹配,GEOIP,US,DIRECT 匹配所有美国IP的流量
FINAL,最终匹配的行为,这个必须放在最后,指定不能在前面任意一个规则匹配的流量行为
注:REJECT 过滤规则结合广告域名匹配可以过滤掉应用程序内部广告,这个真的很棒。
no-resolve
当有 Rule 中有 GEOIP 和 IP-CIDR 配置时,Surge 将对所有基于域名的请求流量进行域名解析代理,通过可选配置no-resolve 可以指定特定流量不运行域名解析代理,如:

GEOIP,US,DIRECT,no-resolve
IP-CIDR,172.16.0.0/12,DIRECT,no-resolve
force-remote-dns
基于 TCP 协议的应用程序在 TUN interface 模式中需要请求服务时,首先会发送域名解析请求,之后再发送 TCP 数据,如果域名无法在本地被解析,可以通过 force-remote-dns 配置域名在远程代理上进行解析,如:

DOMAIN,www.apple.com,Proxy,force-remote-dns
DOMAIN-SUFFIX,apple.com,Proxy,force-remote-dns
DOMAIN-KEYWORD,google,Proxy,force-remote-dns
注意: 这个仅对 TUN interface 有效,对于 Proxy Server 而言,所有的请求的 DNS 都是在远程代理上解析的。
保护配置
所有以#!REQUIRE-PROTECTED开头的配置都是保护配置,保护配置在 Surge 软件中将不会被显示,在你将私人配置发送给其他人共享时可以将私有信息设为保护配置。

注:这其也没啥用,配置文件本身是可读的,下载文件直接看就是了。
URL Scheme
Surge 支持 URL Scheme 满足重度/效率使用者的需求,不过按我估计,大概是作者不想被喷,于是就做了最基本的 URL Scheme,不然怎么会不支持切换配置文件的 URL Scheme,当前 Surge 仅支持三种行为以及一个选项:

行为:start,stop,toggle
surge:///start,使用预先选定的服务器配置开启 Surge 服务
surge:///stop,关闭 Surge 服务
surge:///toggle,打开/关闭 Surge 服务
选项:autoclose=true,当行为执行完毕时,自动退出 Surge 软件 surge:///toggle?autoclose=true

我来解释下微信为何会大量占用信令_sunny_新浪博客

我来解释下微信为何会大量占用信令_sunny_新浪博客: 我来解释下微信为何会大量占用信令 (2014-03-18 13:08:28)转载▼
分类: Android
1)手机和基站间的连接不是QQ或者微信这种应用层软件能够控制的。这些通讯控制功能全部是无线通信协议底层的实现,加密封装在3G基带处理芯片内部;芯片供应商通常是不会开放这些协议的,连手机厂商都动不了;这是保证整个移动通信网络的完整性和安全性。ios操作系统看不见,Android也看不见,所以根本不可能发送什么物理层级别的控制信令,更别说QQ微信这种级别的软件了。
2)手机一旦休眠,cpu断电,包括操作系统在内没有任何程序可以执行,QQ微信更不可能有机会执行。所以什么QQ微信隔段时间就会迫使手机去连基站这种连基本常识都没有的说法就不要提了。
3)休眠时的手机只有3G基带芯片和射频在工作,每隔一定时间通过寻呼信道与基站保持联系。
4)手机QQ微信一直在线实际上是ios/android的推送服务模拟出来的假象,让你以为他一直在线,其实没有。
5)具体来说,就是当你要Q/微信好友时,实际上是腾讯的服务器,通过ios的推送服务器(苹果在全球部署了大量的这种服务器),再经过运营商的网关,向好友的手机发出一条推送信息。
6)网关在解析出对方手机的号码、小区位置等信息后,通过移动基站寻呼信道向对方的手机发出寻呼信令
7)手机的3G基带在接受到寻呼信令后,会触发重新给cpu上电,把手机从休眠状态中唤醒,ios系统从断点恢复加载,整个手机恢复到休眠前状态,恢复与基站的上层数据连接
8)网关在数据连接恢复后把推送信息发送到对方手机上。
9)对方手机接受到这条推送信息后,ios系统内的推送服务子系统解析出这条推送信息是发给QQ/微信的,那么通过系统api启动QQ/微信客户端软件应用。
10)QQ/微信客户端应用软件里面已经注册了监听系统的推送调用,此时收到系统发来的推送系统后,马上重新登陆QQ/微信服务器,恢复休眠前在线状态,从服务器下载好友传来的真正的消息内容,如文字、图片等

Wednesday, December 14, 2016

DNSPod各个套餐的DNS地址-DNSPod 技术支持-DNSPod-免费智能DNS解析服务商-电信_网通_教育网,智能DNS

DNSPod各个套餐的DNS地址-DNSPod 技术支持-DNSPod-免费智能DNS解析服务商-电信_网通_教育网,智能DNS

DNSPod各个套餐的DNS地址
每种域名套餐对应的DNS地址是不同的!

免费DNS地址:f1g1ns1.dnspod.net/f1g1ns2.dnspod.net (对应 10 台服务器);
豪华DNS地址:ns1.dnsv2.com/ns2.dnsv2.com (对应 12 台服务器);
企业I DNS地址:ns1.dnsv3.com/ns2.dnsv3.com (对应 14 台服务器);
企业II DNS地址:ns1.dnsv4.com/ns2.dnsv4.com (对应 18 台服务器);
企业III DNS地址:ns1.dnsv5.com/ns2.dnsv5.com (对应 22 台服务器);
个人专业版DNS地址:ns3.dnsv2.com/ns4.dnsv2.com (对应 12 台服务器);
企业创业版DNS地址:ns3.dnsv3.com/ns4.dnsv3.com (对应 14 台服务器);
企业标准版 DNS地址:ns3.dnsv4.com/ns4.dnsv4.com (对应 18 台服务器);
企业旗舰版 DNS地址:ns3.dnsv5.com/ns4.dnsv5.com (对应 22 台服务器)。

2. 当前域名所对应的DNS地址,会在进入管理域名记录面板后,显示在下图位置:

Anomaly - MySQL Full-text Searches and SQLAlchemy; the Present and a Proposed Future

Anomaly - MySQL Full-text Searches and SQLAlchemy; the Present and a Proposed Future
MySQL Full-text Searches and SQLAlchemy; the Present and a Proposed Future

Addendum: Since the publication of this article we discovered Meng Zhuo's MySQL full-text add-on for SQLAlchemy and have decided to side with this existing effort over rolling our own solution.

MySQL supports an array of full-text search features recommended for performing advanced textual queries on fields of CHAR, VARCHAR and TEXT types. For a variety of great reasons Python is our language of choice and there's no better Pythonic interface to a relational database than SQLAlchemy.

SQLAlchemy is feature-packed and rock-solid. Unfortunately, it does not completely support all of what MySQL has to offer in the way of full-text searches.

The following article provides a brief introduction to full-text searching in MySQL, how—and to what extent—you can use it in SQLAlchemy 1.0.13 as well as our proposal for feature-complete full-text searching in SQLAlchemy and working with the core team to merge it back into the core code base.

Working with Full-text Indexes

Full-text indexes are created on one or a group of fields. The exact same set of fields must be used in MATCH and AGAINST syntax to perform full-text queries.

Indexes can be created as part of the initial table definition:

CREATE TABLE customer (
customer_id INT UNSIGNED AUTO_INCREMENT NOT NULL PRIMARY KEY,
name VARCHAR(200),
physical_address_street TEXT(),
FULLTEXT name_index (name)
) ENGINE=InnoDB;
Or on an existing table:

CREATE FULLTEXT INDEX `name_index` ON customer(name);

/* or across multiple fields */

CREATE FULLTEXT INDEX `fulltext_index` ON customer(name, physical_address_street);
Queries performed using MATCH and AGAINST are able to make use of the full-text index. They can be run in three different modes:

Natural Language
Natural Language with Query Expansion
Boolean Mode
Note: The exact same fields used to create the full-text index must appear when running full-text searches.

MySQL's documentation covers querying in great detail. A basic BOOLEAN mode query would look as follows:

SELECT customer_id FROM customer WHERE MATCH(name) AGAINST('anomaly+ soft*' IN BOOLEAN MODE);

/* or across multiple fields */

SELECT customer_id FROM customer WHERE MATCH(name, physical_address_street)
AGAINST('anomaly+ soft*' IN BOOLEAN MODE);
Full-text Searching via SQLAlchemy

At present SQLAlchemy (as at version 1.0.13) has implemented fairly basic support for full-text searching. The query interface is limited to single fields and queries are preset to run in BOOLEAN mode.

customers = session.query(Customer).filter(Customer.name.match("anomaly")).all()
Running queries against multiple fields or other modes is only possible by rolling your own ClauseElement. There's an extensive thread on StackOverflow where Mike Bayer, the author of SQLAlchemy, has outlined a detailed solution.

Note: Using BOOLEAN mode operators crash SQLAlchemy. This would most likely be caused by the match function failing to compile the operators.

There's also no support for creating full-text indexes via the declarative_base. The best option is to hook onto the after_create event and create the index via the DDL.

event.listen(
Customer.__table__,
"after_create",
DDL("CREATE FULLTEXT INDEX `name` ON %(table)s(name)"))
Proposed Future

While investigating full-text support in SQLAlchemy, I had a brief discussion on the mailing list. As a result we've decided to take on the job of completely implementing MySQL full-text support in SQLAlchemy. We plan to support:

Creation of full-text indexes via the declarative_base
Multiple fields across multiple indexes
Querying multiple fields in BOOLEAN, NATURAL LANGUAGE and NATURAL LANGUAGE mode with QUERY EXPANSION
You can follow the development on our forked repository on Github.

databasepython

Monday, November 28, 2016

导致TCP协议联通性的两类问题汇总_百度文库

导致TCP协议联通性的两类问题汇总_百度文库
导致TCP联通性的两类问题汇总 手机无线网络环境比PC网络环境更复杂,因此也带来了一些特殊的网络问题。这些网络问题并非手机无线网独有,但是在手机无线网络中,由于有无线运营商这个环节,这个环节对互联网运营商是个黑盒,他们运维的一些特殊配置,更可能导致无线网络上的问题。这里主要汇总两类问题。 1:tcp连接无法正常建立 2:大数据传输不正常 这两类问题都不是普遍存在的,可能只存在某个运营商网络,或者某个省份的运营商。 首先来分析第一个问题。 Tcp连接无法建立,最直接的表现是app与server之间的tcp连接不能正常完成,导致用户无法正常连接到server上。这类问题处理起来的确棘手。涉及到的情况比较复杂,比如也许是用户当时的无线信号不稳定,也许是用户连接的wifi掉线了,也许某个运营商与server之间的网络就不通,等等类似问题。这类问题都有共通的几个特性:1,非大面积发生;2,比较难重现;3,用户现场比较难获取;4,用户配合定位问题难度搞;5,用户tcpdump抓包困难。 手机Qzone在运营过程中也遇到了以上问题,像信号不稳定,wifi不通这类问题比较好解决。联系用户,核实情况一般都可以解决。其中有一个隐藏比较深,是由于server内核参数设置不当引起的问题。无线环境下,有很大一部分原因是这个问题导致的。 做互联网server开发,为了支持高并发,server一般都会打开tcp_tw_reuse,tcp_tw_recycle。这两个选项都可以控制tcp快速回收。还有另一个设置 tcp_timestamp。这个设置往往被忽略了。如果tcp_tw_recycle,tcp_timestamp同时被设置,可能就会导致connect连接失败的问题。这里正确做法是关闭 tcp_timestamp。导致这个问题的原因是:某些运营商通过NAT方式组网,如果NAT网络中有台设备的时间戳有问题,就会导致其他机器都不能正常的connect上我们的server了。 现在分析下第二个问题。 第二个问题一般出现在app与server之间传输传输大数据,导致出现丢失数据问题。此类问题一般都是由MTU引起。这里特指PMTU,tcp传输路径上的MTU。因为两端的MTU在tcp三次握手的过程中已经协商好了,PMTU确无法协商。IP包头中有个DF标志,默认情况下是1,标识不允许IP包分片。如果有一个IP包在传输过程中经过一个路由器,这个路由器的MTU设置小于此IP包,此时候由于DF为1,路由器不做IP分片,而是直接丢弃此IP包,并返回一个ICMP报文。(现实中很多路由器设置了不返回ICMP报文,而是直接丢弃IP包)发送端无法感知IP是因为MTU问题被丢弃,还是其他原因未到达对端,对于发送端来说,就是这个IP包丢包,于是触发了tcp的超时重传。但是由于这个问题是由于路由器MTU导致的,重传也解决不了问题。这里解决问题有几个方法: 1:更改server的mtu,改到一个比较小的值。这个方案可行,但是会影响这台server上的所有网络数据的mtu。一般建议改到1400左右,太小可能会引起更多ip碎片。 2:程序中设置DF为0,允许路由器对IP包做分片。 关闭df的方法: int val = IP_PMTUDISC_DONT;

int ret = setsockopt(sock, IPPROTO_IP, IP_MTU_DISCOVER, &val, sizeof(val)); 打开df的方法: int val = IP_PMTUDISC_DO; int ret = setsockopt(sock, IPPROTO_IP, IP_MTU_DISCOVER, &val, sizeof(val)); 3:可以通过程序设置mss,间接控制每个IP包大小。好处是不用更改server的mtu,只影响本程序。用setsockopt, SOL_TCP, TCP_MAXSEG设置。 以上方法主要解决了下行不通,有时候app到server也遇到数据太大的问题。 在这方面,Qzone在实现的时候,在客户端预埋了一个逻辑,如果客户端在上行大数据不通的时候,可以在客户端设置mss大小。具体设置多大,可以通过配置来控制。

Thursday, November 3, 2016

DNS�NO�GLUE_flyfish_carol_新浪博客

DNS�NO�GLUE_flyfish_carol_新浪博客





DNS NO GLUE

 (2006-12-11 15:22:31)

DNS NO GLUE

Author: Hagen.GoO  转载请注明作者出处
MSN contact: wantm009@hotmail.com
Keyword: DNS Report,NameServer,NS RRs,NO-GLUE,

    DNS Report http://www.dnsreport.com/ 的在线工具在校验某域名配置时,parent nameservers 部分老是有 Glue at parent nameservers 即 NO GLUE 的警告。
    看了网页上的简短说明,也不是非常明白所谓的 NO GLUE 到底是什么概念。翻阅相关书籍,略有斩获,招贴备忘。
    我们知道,按照ICANN/IANA的协调:
        COM. NET. 2个域的 nameservers 是 a.gtld-servers.net. b.gtld-servers.net. 等 A-M 13台服务器。
        CN 和 COM.CN. 等中国国家的ccTLDs 的 nameservers 是 a.dns.cn. b.cns.cn. 等6台服务器。
    第一种情况:比如我们申请到域名 domain.com.,在域名配置的控制面板可以注册并添加多条 NS RRs,如 ns1.domain.com. ns2.domain.com.等。这些新添加的 NS RRs 在其 parent nameservers 将以如下的方式存在:
domain.com.       IN  NS    ns1.domain.com.
domain.com.       IN  NS    ns2.domain.com.
ns1.domain.com.   IN   A    xxx.xxx.xxx.xxx
ns2.domain.com.   IN   A    xxx.xxx.xxx.xxx
    即,我们可以使用类似这样的命令:
        nslookup -q=any ns1.domain.com. a.gtld-servers.net.
        nslookup -q=ns domain.com. a.gtld-servers.net.
    直接在 a.gtld-servers.net. 处查询(非递归)到 domain.com. 相关的 NS RRs 以及对应的IP地址,这种情况就是 GLUE 的。

    第二种情况:比如我们申请到域名 domain.com.,我们使用域名注册商提供的,且已经存在的 NS 来负责解析此域名,比如中国万网的 dns7.hichina.com. dns8.hichina.com.,这个时候在其 parent nameservers 将以如下的方式存在:
domain.com.       IN  NS    dns7.hichina.com.
domain.com.       IN  NS    dns8.hichina.com.
    由于这是2台已经注册过的 nameservers , 即还有:
dns7.hichina.com.    IN   A    xxx.xxx.xxx.xxx
dns8.hichina.com.    IN   A    xxx.xxx.xxx.xxx
    dns7.hichina.com. dns8.hichina.com.和 domain.com. 同属于一个 TLDs,所以我们在向 a.gtld-servers.net. 查询(非递归) domain.com. 相关 NS RRs以及对应IP地址时,a.gtld-servers.net. 都能给出正确应答,这种情况也是 GLUE 的。

    第三种情况:比如我们申请到中国国家域名 newdomain.cn.,我们使用域名注册商提供的 NS 来负责解析此域名,比如中国万网的 dns7.hichina.com. dns8.hichina.com.,这时 domain.cn.的相关NS在其 parent nameservers a.dns.cn中将以如下的方式存在:
newdomain.cn.       IN  NS    dns7.hichina.com.
newdomain.cn.       IN  NS    dns8.hichina.com.
    由于 dns7.hichina.com. dns8.hichina.com.和 newdomain.cn. 不是同属于一个 TLDs,所以我们在向 a.dns.cn. 查询(非递归) newdomain.cn. 相关 NS RRs以及对应IP地址时,a.dns.cn. 只都能给出 NS 记录,不能给出其对应 IP 地址,需要我们再另外一次的递归查询才能得到对应 IP 地址。这种情况就是 NO GLUE 的。

    另外还有一种特殊的情况,类似第二种情况,目标域名和 nameservers 的所属域名是同一个 TLDs 的,比如都是 COM. 的,但这几个 nameservers 没有在所属域名的 parent nameservers 中注册过,那么这种情况也应当是 NO GLUE 的。
   
    总结一句话,在同一个 TLDs 的 parent namservers 上直接可以查询到目标域名相关的 NS RRs 以及 NS 对应的 IP 地址的情况就是 GLUE ,否则就是 NO GLUE。虽然 NO GLUE 略微会造成解析上的延迟,但不是致命的问题。有条件的域名建议使用 GLUE 的 NS。
    另外: DNS Report 的工具在诊断 COM.CN.中国国家域名是否 GLUE 上有一点小小的问题,实用的时候,可以斟酌看待。

Wednesday, October 12, 2016

Python in Xcode 7 - Stack Overflow

?? how can run with "Debug executable" on?

Python in Xcode 7 - Stack Overflow


244
down vote
accepted
I figured it out! The steps make it look like it will take more effort than it actually does.

These instructions are for creating a project from scratch. If you have existing Python scripts that you wish to include in this project, you will obviously need to slightly deviate from these instructions.

If you find that these instructions no longer work or are unclear due to changes in Xcode updates, please let me know. I will make the necessary corrections.

Open Xcode. The instructions for either are the same.
In the menu bar, click “File” → “New” → “New Project…”.
Select “Other” in the left pane, then "External Build System" in the right page, and next click "Next".
Enter the product name, organization name, or organization identifier.
For the “Build Tool” field, type in /usr/local/bin/python3 for Python 3 or /usr/bin/python for Python 2 and then click “Next”. Note that this assumes you have the symbolic link (that is setup by default) that resolves to the Python executable. If you are unsure as to where your Python executables are, enter either of these commands into Terminal: which python3 and which python.
Click “Next”.
Choose where to save it and click “Create”.
In the menu bar, click “File” → “New” → “New File…”.
Select “Other” under “OS X”.
Select “Empty” and click “Next”.
Navigate to the project folder (it will not work, otherwise), enter the name of the Python file (including the “.py” extension), and click “Create”.
In the menu bar, click “Product” → “Scheme” → “Edit Scheme…”.
Click “Run” in the left pane.
In the “Info” tab, click the “Executable” field and then click “Other…”.
Navigate to the executable from Step 6. You may need to use ⇧⌘G to type in the directory if it is hidden.
Select the executable and click "Choose".
Uncheck “Debug executable”. If you skip this step, Xcode will try to debug the Python executable itself. I am unaware of a way to integrate an external debugging tool into Xcode.
Click the “+” icon under “Arguments Passed On Launch”. You might have to expand that section by clicking on the triangle pointing to the right.
Type in $(SRCROOT)/ (or $(SOURCE_ROOT)/) and then the name of the Python file you want to test. Remember, the Python program must be in the project folder. Otherwise, you will have to type out the full path (or relative path if it's in a subfolder of the project folder) here. If there are spaces anywhere in the full path, you must include quotation marks at the beginning and end of this.
Click “Close”.
Note that if you open the "Utilities" panel, with the "Show the File inspector" tab active, the file type is automatically set to "Default - Python script". Feel free to look through all the file type options it has, to gain an idea as to what all it is capable of doing. The method above can be applied to any interpreted language. As of right now, I have yet to figure out exactly how to get it to work with Java; then again, I haven't done too much research. Surely there is some documentation floating around on the web about all of this.

Running without administrative privileges:

If you do not have administrative privileges or are not in the Developer group, you can still use Xcode for Python programming (but you still won't be able to develop in languages that require compiling). Instead of using the play button, in the menu bar, click "Product" → "Perform Action" → "Run Without Building" or simply use the keyboard shortcut ^⌘R.

Other Notes:

To change the text encoding, line endings, and/or indentation settings, open the "Utilities" panel and click "Show the File inspector" tab active. There, you will find these settings.

For more information about Xcode's build settings, there is no better source than this. I'd be interested in hearing from somebody who got this to work with unsupported compiled languages. This process should work for any other interpreted language. Just be sure to change Step 6 and Step 16 accordingly.

Wednesday, October 5, 2016

A Celery-like Python Task Queue in 55 Lines of Code

A Celery-like Python Task Queue in 55 Lines of Code: A Celery-like Python Task Queue in 55 Lines of Code



A Celery-like Python Task Queue in 55 Lines of Code

Update: you can check this out on GitHub here.
Celery is probably the best known task queuing Python package around. It makes asynchronous execution of Python code both possible and reasonably straightforward. It does, however, come with a good deal of complexity, and it's not as simple to use as I would like (i.e. for many use cases it's overkill). So I wrote a distributed Python task queue. In 55 lines of code (caveat: using two awesome libraries).

Background

What does a distributed task queue do? It takes code like the following (taken from the documentation forRQ, another Celery alternative):
import requests

def count_words_in_page(url):
    resp = requests.get(url)
    return len(resp.text.split())
and allows it to be sent to a worker process (possibly on another machine) for execution. The worker process then sends back the results after the calculation is complete. In the meantime, the sender doesn't have to block waiting for the (possibly expensive) calculation to complete. They can just periodically check if the results are ready.
So what's the absolute simplest way we could do this? I submit to you, brokest.py:
"""Broker-less distributed task queue."""
import pickle

import zmq
import cloud

HOST = '127.0.0.1'
PORT = 9090
TASK_SOCKET = zmq.Context().socket(zmq.REQ)
TASK_SOCKET.connect('tcp://{}:{}'.format(HOST, PORT))

class Worker(object):
    """A remote task executor."""

    def __init__(self, host=HOST, port=PORT):
        """Initialize worker."""
        self.host = host
        self.port = port
        self._context = zmq.Context()
        self._socket = self._context.socket(zmq.REP)

    def start(self):
        """Start listening for tasks."""
        self._socket.bind('tcp://{}:{}'.format(self.host, self.port))
        while True:
            runnable_string = self._socket.recv_pyobj()
            runnable = pickle.loads(runnable_string)
            self._socket.send_pyobj('')
            args = self._socket.recv_pyobj()
            self._socket.send_pyobj('')
            kwargs = self._socket.recv_pyobj()
            response = self._do_work(runnable, args, kwargs)
            self._socket.send_pyobj(response)

    def _do_work(self, task, args, kwargs):
        """Return the result of executing the given task."""
        print('Running [{}] with args [{}] and kwargs [{}]'.format(
            task, args, kwargs))
        return task(*args, **kwargs)

def queue(runnable, *args, **kwargs):
    """Return the result of running the task *runnable* with the given 
    arguments."""
    runnable_string = cloud.serialization.cloudpickle.dumps(runnable)
    TASK_SOCKET.send_pyobj(runnable_string)
    TASK_SOCKET.recv()
    TASK_SOCKET.send_pyobj(args)
    TASK_SOCKET.recv()
    TASK_SOCKET.send_pyobj(kwargs)
    results = TASK_SOCKET.recv_pyobj()
    return results

if __name__ == '__main__':
    w = Worker()
    w.start()
And to use it? Here's the complete contents of app.py:
import requests
from brokest import queue

def count_words_in_page(url):
    resp = requests.get(url)
    return len(resp.text.split())

result = queue(count_words_in_page, 'http://www.jeffknupp.com')
print result
Rather than calling the function directly, you simply call brokest.queue with the function and it arguments as arguments. While the current implementation is blocking, it would be trivially easy to make it non-blocking. Adding multiple workers is just a matter of adding code to make use of a config file with their locations.
Clearly, the stars here are zmq and cloudZeroMQ makes creating distributed systems a lot easier, and the documentation is chock full of ZeroMQ design patterns (ours is probably the simplest one, Request-Reply).
cloud is the LGPL'd PiCloud library. PiCloud is a company whose value proposition is that they let you seamlessly run computationally intensive Python code using Amazon EC2 for computing resources. Part of making that a reality, though, required a way to pickle functions and their dependencies (functions are not normally directly pickle-able). In our example, the Worker is able to make use of code using requestslibrary despite not having imported it. It's the secret sauce that makes this all possible.

Is This For Real?

The code works, but my intention was not to create a production quality distributed task queue. Rather, it was to show how new libraries are making it easier than ever to create distributed systems. Having a way to pickle code objects and their dependencies is a huge win, and I'm angry I hadn't heard of PiCloud earlier.
One of the best things about being a programmer is the ability to tinker not just with things, but with ideas. I can take existing ideas and tweak them, or combine existing ideas in new ways. I think brokest is an interesting example of how easy it has become to create distributed systems.
Posted on  by 

Discuss Posts With Other Readers at discourse.jeffknupp.com!

Like this article?

Why not sign up for Python Tutoring? Sessions can be held remotely using Google+/Skype or in-person if you're in the NYC area. Email jeff@jeffknupp.com if interested.
Sign up for the free jeffknupp.com email newsletter. Sent roughly once a month, it focuses on Python programming, scalable web development, and growing your freelance consultancy. And of course, you'll never be spammed, your privacy is protected, and you can opt out at any time.

Wednesday, August 17, 2016

A successful Git branching model � nvie.com

A successful Git branching model � nvie.com
A successful Git branching model
By Vincent Driessen
on Tuesday, January 05, 2010

We Provide the Highest Level of Transparency & Control for Cloud
ads via Carbon
In this post I present the development model that I’ve introduced for some of my projects (both at work and private) about a year ago, and which has turned out to be very successful. I’ve been meaning to write about it for a while now, but I’ve never really found the time to do so thoroughly, until now. I won’t talk about any of the projects’ details, merely about the branching strategy and release management.



It focuses around Git as the tool for the versioning of all of our source code. (By the way, if you’re interested in Git, our company GitPrime provides some awesome realtime data analytics on software engineering performance.)

Why git?
For a thorough discussion on the pros and cons of Git compared to centralized source code control systems, see the web. There are plenty of flame wars going on there. As a developer, I prefer Git above all other tools around today. Git really changed the way developers think of merging and branching. From the classic CVS/Subversion world I came from, merging/branching has always been considered a bit scary (“beware of merge conflicts, they bite you!”) and something you only do every once in a while.

But with Git, these actions are extremely cheap and simple, and they are considered one of the core parts of your daily workflow, really. For example, in CVS/Subversion books, branching and merging is first discussed in the later chapters (for advanced users), while in every Git book, it’s already covered in chapter 3 (basics).

As a consequence of its simplicity and repetitive nature, branching and merging are no longer something to be afraid of. Version control tools are supposed to assist in branching/merging more than anything else.

Enough about the tools, let’s head onto the development model. The model that I’m going to present here is essentially no more than a set of procedures that every team member has to follow in order to come to a managed software development process.

Decentralized but centralized
The repository setup that we use and that works well with this branching model, is that with a central “truth” repo. Note that this repo is only considered to be the central one (since Git is a DVCS, there is no such thing as a central repo at a technical level). We will refer to this repo as origin, since this name is familiar to all Git users.



Each developer pulls and pushes to origin. But besides the centralized push-pull relationships, each developer may also pull changes from other peers to form sub teams. For example, this might be useful to work together with two or more developers on a big new feature, before pushing the work in progress to origin prematurely. In the figure above, there are subteams of Alice and Bob, Alice and David, and Clair and David.

Technically, this means nothing more than that Alice has defined a Git remote, named bob, pointing to Bob’s repository, and vice versa.

The main branches


At the core, the development model is greatly inspired by existing models out there. The central repo holds two main branches with an infinite lifetime:

master
develop
The master branch at origin should be familiar to every Git user. Parallel to the master branch, another branch exists called develop.

We consider origin/master to be the main branch where the source code of HEAD always reflects a production-ready state.

We consider origin/develop to be the main branch where the source code of HEAD always reflects a state with the latest delivered development changes for the next release. Some would call this the “integration branch”. This is where any automatic nightly builds are built from.

When the source code in the develop branch reaches a stable point and is ready to be released, all of the changes should be merged back into master somehow and then tagged with a release number. How this is done in detail will be discussed further on.

Therefore, each time when changes are merged back into master, this is a new production release by definition. We tend to be very strict at this, so that theoretically, we could use a Git hook script to automatically build and roll-out our software to our production servers everytime there was a commit on master.

Supporting branches
Next to the main branches master and develop, our development model uses a variety of supporting branches to aid parallel development between team members, ease tracking of features, prepare for production releases and to assist in quickly fixing live production problems. Unlike the main branches, these branches always have a limited life time, since they will be removed eventually.

The different types of branches we may use are:

Feature branches
Release branches
Hotfix branches
Each of these branches have a specific purpose and are bound to strict rules as to which branches may be their originating branch and which branches must be their merge targets. We will walk through them in a minute.

By no means are these branches “special” from a technical perspective. The branch types are categorized by how we use them. They are of course plain old Git branches.

Feature branches


May branch off from:
develop
Must merge back into:
develop
Branch naming convention:
anything except master, develop, release-*, or hotfix-*
Feature branches (or sometimes called topic branches) are used to develop new features for the upcoming or a distant future release. When starting development of a feature, the target release in which this feature will be incorporated may well be unknown at that point. The essence of a feature branch is that it exists as long as the feature is in development, but will eventually be merged back into develop (to definitely add the new feature to the upcoming release) or discarded (in case of a disappointing experiment).

Feature branches typically exist in developer repos only, not in origin.

Creating a feature branch
When starting work on a new feature, branch off from the develop branch.

$ git checkout -b myfeature develop
Switched to a new branch "myfeature"
Incorporating a finished feature on develop
Finished features may be merged into the develop branch to definitely add them to the upcoming release:

$ git checkout develop
Switched to branch 'develop'
$ git merge --no-ff myfeature
Updating ea1b82a..05e9557
(Summary of changes)
$ git branch -d myfeature
Deleted branch myfeature (was 05e9557).
$ git push origin develop
The --no-ff flag causes the merge to always create a new commit object, even if the merge could be performed with a fast-forward. This avoids losing information about the historical existence of a feature branch and groups together all commits that together added the feature. Compare:



In the latter case, it is impossible to see from the Git history which of the commit objects together have implemented a feature—you would have to manually read all the log messages. Reverting a whole feature (i.e. a group of commits), is a true headache in the latter situation, whereas it is easily done if the --no-ff flag was used.

Yes, it will create a few more (empty) commit objects, but the gain is much bigger than the cost.

Release branches
May branch off from:
develop
Must merge back into:
develop and master
Branch naming convention:
release-*
Release branches support preparation of a new production release. They allow for last-minute dotting of i’s and crossing t’s. Furthermore, they allow for minor bug fixes and preparing meta-data for a release (version number, build dates, etc.). By doing all of this work on a release branch, the develop branch is cleared to receive features for the next big release.

The key moment to branch off a new release branch from develop is when develop (almost) reflects the desired state of the new release. At least all features that are targeted for the release-to-be-built must be merged in to develop at this point in time. All features targeted at future releases may not—they must wait until after the release branch is branched off.

It is exactly at the start of a release branch that the upcoming release gets assigned a version number—not any earlier. Up until that moment, the develop branch reflected changes for the “next release”, but it is unclear whether that “next release” will eventually become 0.3 or 1.0, until the release branch is started. That decision is made on the start of the release branch and is carried out by the project’s rules on version number bumping.

Creating a release branch
Release branches are created from the develop branch. For example, say version 1.1.5 is the current production release and we have a big release coming up. The state of develop is ready for the “next release” and we have decided that this will become version 1.2 (rather than 1.1.6 or 2.0). So we branch off and give the release branch a name reflecting the new version number:

$ git checkout -b release-1.2 develop
Switched to a new branch "release-1.2"
$ ./bump-version.sh 1.2
Files modified successfully, version bumped to 1.2.
$ git commit -a -m "Bumped version number to 1.2"
[release-1.2 74d9424] Bumped version number to 1.2
1 files changed, 1 insertions(+), 1 deletions(-)
After creating a new branch and switching to it, we bump the version number. Here, bump-version.sh is a fictional shell script that changes some files in the working copy to reflect the new version. (This can of course be a manual change—the point being that some files change.) Then, the bumped version number is committed.

This new branch may exist there for a while, until the release may be rolled out definitely. During that time, bug fixes may be applied in this branch (rather than on the develop branch). Adding large new features here is strictly prohibited. They must be merged into develop, and therefore, wait for the next big release.

Finishing a release branch
When the state of the release branch is ready to become a real release, some actions need to be carried out. First, the release branch is merged into master (since every commit on master is a new release by definition, remember). Next, that commit on master must be tagged for easy future reference to this historical version. Finally, the changes made on the release branch need to be merged back into develop, so that future releases also contain these bug fixes.

The first two steps in Git:

$ git checkout master
Switched to branch 'master'
$ git merge --no-ff release-1.2
Merge made by recursive.
(Summary of changes)
$ git tag -a 1.2
The release is now done, and tagged for future reference.

Edit: You might as well want to use the -s or -u flags to sign your tag cryptographically.

To keep the changes made in the release branch, we need to merge those back into develop, though. In Git:

$ git checkout develop
Switched to branch 'develop'
$ git merge --no-ff release-1.2
Merge made by recursive.
(Summary of changes)
This step may well lead to a merge conflict (probably even, since we have changed the version number). If so, fix it and commit.

Now we are really done and the release branch may be removed, since we don’t need it anymore:

$ git branch -d release-1.2
Deleted branch release-1.2 (was ff452fe).
Hotfix branches


May branch off from:
master
Must merge back into:
develop and master
Branch naming convention:
hotfix-*
Hotfix branches are very much like release branches in that they are also meant to prepare for a new production release, albeit unplanned. They arise from the necessity to act immediately upon an undesired state of a live production version. When a critical bug in a production version must be resolved immediately, a hotfix branch may be branched off from the corresponding tag on the master branch that marks the production version.

The essence is that work of team members (on the develop branch) can continue, while another person is preparing a quick production fix.

Creating the hotfix branch
Hotfix branches are created from the master branch. For example, say version 1.2 is the current production release running live and causing troubles due to a severe bug. But changes on develop are yet unstable. We may then branch off a hotfix branch and start fixing the problem:

$ git checkout -b hotfix-1.2.1 master
Switched to a new branch "hotfix-1.2.1"
$ ./bump-version.sh 1.2.1
Files modified successfully, version bumped to 1.2.1.
$ git commit -a -m "Bumped version number to 1.2.1"
[hotfix-1.2.1 41e61bb] Bumped version number to 1.2.1
1 files changed, 1 insertions(+), 1 deletions(-)
Don’t forget to bump the version number after branching off!

Then, fix the bug and commit the fix in one or more separate commits.

$ git commit -m "Fixed severe production problem"
[hotfix-1.2.1 abbe5d6] Fixed severe production problem
5 files changed, 32 insertions(+), 17 deletions(-)
Finishing a hotfix branch

When finished, the bugfix needs to be merged back into master, but also needs to be merged back into develop, in order to safeguard that the bugfix is included in the next release as well. This is completely similar to how release branches are finished.

First, update master and tag the release.

$ git checkout master
Switched to branch 'master'
$ git merge --no-ff hotfix-1.2.1
Merge made by recursive.
(Summary of changes)
$ git tag -a 1.2.1
Edit: You might as well want to use the -s or -u flags to sign your tag cryptographically.

Next, include the bugfix in develop, too:

$ git checkout develop
Switched to branch 'develop'
$ git merge --no-ff hotfix-1.2.1
Merge made by recursive.
(Summary of changes)
The one exception to the rule here is that, when a release branch currently exists, the hotfix changes need to be merged into that release branch, instead of develop. Back-merging the bugfix into the release branch will eventually result in the bugfix being merged into develop too, when the release branch is finished. (If work in develop immediately requires this bugfix and cannot wait for the release branch to be finished, you may safely merge the bugfix into develop now already as well.)

Finally, remove the temporary branch:

$ git branch -d hotfix-1.2.1
Deleted branch hotfix-1.2.1 (was abbe5d6).
Summary
While there is nothing really shocking new to this branching model, the “big picture” figure that this post began with has turned out to be tremendously useful in our projects. It forms an elegant mental model that is easy to comprehend and allows team members to develop a shared understanding of the branching and releasing processes.

A high-quality PDF version of the figure is provided here. Go ahead and hang it on the wall for quick reference at any time.

Update: And for anyone who requested it: here’s the gitflow-model.src.key of the main diagram image (Apple Keynote).


Git-branching-model.pdf

If you want to get in touch, I'm @nvie on Twitter.

Thursday, July 28, 2016

version control - How to remove/delete a large file from commit history in Git repository? - Stack Overflow

version control - How to remove/delete a large file from commit history in Git repository? - Stack Overflow: active oldest votes
up vote
207
down vote
accepted
Use the BFG Repo-Cleaner, a simpler, faster alternative to git-filter-branch specifically designed for removing unwanted files from Git history.

Carefully follow the usage instructions, the core part is just this:

$ java -jar bfg.jar --strip-blobs-bigger-than 100M my-repo.git
Any files over 100MB in size (that aren't in your latest commit) will be removed from your Git repository's history. You can then use git gc to clean away the dead data:

$ git gc --prune=now --aggressive
The BFG is typically at least 10-50x faster than running git-filter-branch, and generally easier to use.

Full disclosure: I'm the author of the BFG Repo-Cleaner.

Tuesday, May 31, 2016

Mac OS X Reversing Tools | Reverse Engineering Mac OS X

Mac OS X Reversing Tools | Reverse Engineering Mac OS X: Tools
This page will hold local copies of reversing tools and scripts useful for Mac OS X reversing.

gdbinit – enhanced gdb output

0xEd v1.0.7 – hex editor
(SHA1(0xED.tar.bz2)= f64466b2d3cbf7b6d64eccfc1a36f8c0a7e3866d)

HexFiend – another hex editor
(SHA1(HexFiend.dmg)= 690ac9f60ab85ec6430b3db0376d0d20d3cecd9a)

Synalize it v1.0.3 – hex editor with binary file analysis grammar (looks great!!!) – Original website
(SHA256(SynalyzeIt_1.0.3.1.zip)= ab71d0f2e573321946ec144e60594d4155961b42aeafb2f5b5080bf9961348d0)

OTX v0.16b – disassembler
(SHA1(otx.dmg)= ff4987b7f22da6b289ee2bc7daa7c1a3db64ffed)

offset1.3.pl.gz – my offset calculator for fat binaries
(SHA256(offset1.3.pl.gz)= 2b091f2ea5fddce3ca22251b8d81578ba708811d4a3d2fdce8ae0c8a7972f1b3)

ptool1.3.pl.gz – sort of replacement for otool to display mach-o binaries headers
(SHA256(ptool1.3.pl.gz)= 715481e62978c183ccd82311acb6ccced2d12cab76a0c9ffb0345d653bce37ba)

ocalc.c – ghalen’s offset calculator for fat binaries
(SHA1(ocalc.c)= e32da310af2a25a09fc2de9c4826b113ab8ac705)

onyx-the-black-cat.v0.3 – anti anti-debug kernel module
(SHA1(onyx-the-black-cat-v0-3.tgz)= 194c2e7481113b562c6e23a2b5059769bc9e8ffb)

onyx-the-black-cat-v0.4 – version for Snow Leopard (not 64bit compatible, yet!)
(SHA1(onyx-the-black-cat-v0.4.tgz)= 5dff3c4a9246f2886b470aa0ab60b5e237ca3659)

AlanQuatermain-appencryptor – encryptor/decryptor for Apple Encrypted Binaries
SHA1(AlanQuatermain-appencryptor-a3da7c5.tar.gz)= 3c7f70fed359b7e259f08d00001ead936baef041