Jun Xiao, polished by Anfernee with a couple of bug fixes and enhancement.
router v2 decouples control and data path to some extent, it's a logic router
including both nginx and uls (upstream location server) parts.
uls (aka control plane) is based on legacy router:
- all communications with nats are kept the same as before
- client facing interface (actually connected to nginx) is simplified to
handle the location query/stats update from nginx only.
- the original app facing interface is totally removed.
nginx (aka data plane)
- package lua in nginx so most functionalities can be implemented with it.
- for each request, nginx will generate a subrequest (to fit in nginx
single thread event model) which carries location query/stats info to uls,
the response of the subrequest will return backend addr and also an
opaque tags (only known by uls), and the corresponding main request will
also be awaked and routed to the backend server, when a response from backend
is returned back, the counter associated with the particular opaque tags
will be incremented accoring to the response code. The next incoming
request will always help to carry not-synced stats to uls along with
its location query.
- sticky session and trace header are also implemented here,
also their key related config are moved to nginx part.
testing
- "rake spec" will only test uls, while "rake test" will run both unit test and
integration test with nginx
- integration test requires nginx with lua module installed running
Change-Id: I9b25b24c449b0421754e541ab5ba60a00c551da9
The current recipes seem to work with chef-solo version 0.10.4 but version
0.10.6+ needs unique names for the bash code blocks.
Change-Id: I282abaad0f7a40ec2e7f0de1663da735e839104e
Added a new library function cf_ruby_install that abstracts the ruby
installation. The current recipes were a hack.
Also fixed rake db:migrate to use bundle exec in dev_setup/bin/vcap.
Note for reviewers, This new library function is a straight copy of the
existing code in ruby/recipes/default.rb with the ruby versions and paths passed
in as parameters i.e. No new real code was added, existing code was moved and
abstracted in a more chef compliant manner.
Testing Done: Deployed a single host cloud.
Change-Id: I0d9ba26e0fb5abc6ebe8add00c857faf2e797c26
The existing implementation of 'local_register_only' had a security hole.
Moreover, I'm not sure that it's possible to implement the intent of 'local_register_only'
in Rails/EM safely. We would need to grab the socket for the current request and call getpeername()
on it.
We attempt to provide similar functionality to 'local_register_only' by allowing users to disable
registration and provide seed users to be created at startup. The seed users can then be used to
register additional users.
Test plan:
- New unit tests pass
- Ran CC locally with/without allow_registration set and verified the correct behaviour with vmc.
- BVTs pass locally
- BVTs pass on my deployment
Change-Id: Ib1c96304f7a7364b913a8a09f7178f85857cd46d
We have been using these chef recipes internally at AppFog to deploy our PHP
environment and wanted to share.
Other points of interest:
* Gems built by staging's Rakefile are gitignored
* Removed an errant PHP staging spec from the cloud_controller tree
* Cloud Controller's welcome message and database adapter are configurable
* Some DEA configuration values are now configurable by chef (defaults are
* preserved)
(Note: this is a squashed commit of pull request 118 with
some edits by CF.com (e.g. add the ccdb adapter attribute))
Change-Id: I619d83e2898994714e1c4972ff9ae360d03b4d79
If the ENV variable CLOUD_FOUNDRY_CONFIG_PATH is defined, config files are
picked up from that location. This allows components installed by the chef scripts to be
run by hand. i.e. without using the wrapper vcap/dev_setup/bin/vcap_dev <start/stop>
Change-Id: Ib6c971c6642bcb3b24186169e1c768ee70ba8d7b
This commit fixes a but that happens if you run vcap_dev
with a -d option but not a -n option. (The old
code would leave options["name"] uninitialized)
Change-Id: I820a7fd5ce10f11af96b15f970d3a0b358fec49c
Change-Id: Ib81996a84b0283fd6afc7bb0109d12c620bbcce8
- Add neo4j support
- Save the last deployment in a deployment target file under $HOME
- Change service name i.e. make mysql => mysql_node etc
- Point out the profile file at the end of a deployment, has paths to run vmc
- Audit admin list
- Remove some packages from essentials list
- Increase timeout in bin/vcap that checks if a component is started
patch 3 - Fixed olegs comments.
Testing Done:
Ran Neo4j feature bvt test
Ran single host deployment
Ran multi host mongodb deployment
Change-Id: Id96757bfe109af2f89b9a25e35a669a41c2d259c
Recent change the dev_setup cookbook this change fixex the error.
e.g:
in mysql_node
vcap_logging-0.1.0/lib/vcap/logging.rb:82:in `setup_from_config': undefined method `[]' for nil:NilClass (NoMethodError)
Change-Id: I7706fc01e1a1743bb27de60cf8f89d182f4c18bf
These are my stabs at describing these fields, dont be too surprised if some of
the descriptions are somewhat off from the real use.
In addition to these I looked at the service node and gateway files but they dont seem
to have many interesting fields that need specific descriptions.
patch 3 - know -> known (Thansk Oleg)
patch 4 - added the same comments to files under vcap/{cloud_controller,health_manager,router,dea}
patch 5 - Fixed review comments.
Change-Id: I65185ddc6c08eecf4516944b960367dcb9b1011d
Also updated the deployment readme to document the currently supported multihost
setup.
patch 3 - fix review comments
patch 4 - Fix the readme with the right paths for custom deployments
patch 5 - Fix Vadims comments
patch 6 - Replace 'o' -> '-' :)
Change-Id: Ie2f330248d0c23b900925b0ee54b6e366843713d
* changes:
Add optional profile.sh file to ease deployment
Fixed rvm problem with root installation, and removed the "Fixing init scripts" block because it's already fixed in rvm.
o vcap stop used to fail trying to load config file for components that were
never running on this host
o Start cloud controller before the health manager.
Change-Id: I103685396e332faf0a7eeb9173e2ca6721fdd744
In addtion this change also includes the following changes.
o Remove all backup service related scripts/templates.
o Simple fix to change the deployment name for mulithost mongodb deployment.
Testing Done: verified single host deployment, checked presence of log files in
the new location.
Change-Id: I2b96246da62fe44a3e7de5ca86ca0bf49a20b9c3
Also bumped up the verision of rubygems and bundler.
Testing Done: tested single host setup. Will test multihost setup before
submitting the change.
Change-Id: Ib7ab913f5bf27e90962b5bade2a9b9a7b51cecfd
Tested my changes with cc and services configured to go against my dev23 setup
i.e. "appcloud23.dev.mozycloud.com"
Ran BVTs.
Change-Id: I4e3c97a06c4348c31d610957af6e74cf953ad5c5
Fix for issue cf-135. Usually the pakcage command does the right thing but that
would only work if the same version of java is installed, if openjdk or some
other flavour is installed the package command fails misreably in this case
because it cant find the right repo.
Change-Id: I1aef325e557d338141e68dbd91a070b438f62563
bin/vcap - fix a bug where the specified config file was not being used to
lookup properties.
bin/vcap_dev_setup - check for git clone failures
job_manager.rb - remove postgresql as a services
patch 2 - fix $configdir => $config_dir
Change-Id: Ie0214ce340f62f585f6233a7c7de1e78cc6e1ad8
The deployment config file doesnt need to be an .erb file anymore. Removed some
old code that was using it as an erb file. Also remove the old config samples in
deployments/multihost.
patch 3
Added: sample config files for multihost redis
patch 4
Remove more old/outdated sample files
patch 5
Hit myself on the head.(Added white space remover to vi)
Change-Id: I402e5de08eb9a1fa26c6489ca138d5da7229cf14
Change
This review has changes related to the following 3 things.
1. Move all the default config (that was embeded in
vcap_dev/vcap_dev_setup/job_manager) into chef cookbooks.
2. Add postgres CCDB.
3. Multi host setup for services and dea. This meant separating service
gateway from service nodes. To this effect we added a service gateway role.
Overall a lot of files have changed, most of those changes are related to
moving all the default config values from the various wrapper scripts to the
cookbooks, especially the new cookbook called deployment which is the holding
place for all the deployment related config options.
* dev_setup/bin/vcap:
Added vcap to dev_setup/bin instead of just changing bin/vcap. The reason for
this is that opensource CF code will not be in sync with the
private repo and internal testing of CF dev_setup scrips would not work if it
relied on the version of vcap in the opensource repo i.e. bin/vcap. So now we
just package vcap with dev_setup.
* vcap_dev
* Since all defaults are now maintained in chef scripts. The chef "deployment"
* role/recipe creates a deployment info file that is consumed by this vcap_dev
* script.
* The vcap_dev_setup file now saves the list of components that were installed
for a deployment. This script only starts the components included in that
list.
* Uses the vcap binary from dev_setup/bin
* vcap_dev_setup
Moved a bunch of directory creating code to the deployment cookbook
* Added a CCDB role/recipe.
* Creates and Configures the CC postgres database.
* Added deployment cookbook
* Moves all the comon directory creation code here. Note, we should go back
and see if we really do need all these various directories and probably come
up with a well designed directory layout.
Testing:
* Ran BVTS. Lift BVT failure needs investigating (I will fix/get to the bottom
of it before i submit this change)
* Tested the following multi host setup.
1. Ran mysql database with mysql node on 2 VMs. On a 3rd VM ran everything
else. Verified that BVTs passed.
2. Ran dea on 2 different VMs. Ran everything else on a 3rd VM. Verified
that BVTs passed.
Note: I have only tested the multihost setup with mysql. I will test mongodb
and redis before submitting this change, I expect those to run without any
glitches.
Change-Id: I6a084be09a81bf920eebc62be8d7aa6625cc17e9
All deployments (multi host and single host) are driven through a templated
config file. Look at dev_setup/deployments/sample* for an example of what this
config file looks like. There is sufficient documentation inside the configs.
But at a high level the user specifes "jobs" to "install" and jobs that are
"installed". The "installed" jobs have properties associated with them which are
used by jobs that are in the "install" list. e.g. nats could be an installed job
and its properties like host/port will be used by the jobs in the install list.
The deployment code now goes through this config file, does sanity checking to
verify valid job names, valid properties etc. It leverages "rake" to manage
dependencies between jobs, look at dev_setup/lib/job_*. So we enforce that all
dependent jobs for a given job are either in the "installed" list or in the
"install" list. Once all job specs are verified, we generate the chef runlist
on the fly and the deployment proceeds to install the required components using
chef.
NOTE:
1. For now, multi host service deployment is limited to only selecting a service
say "redis" and the scripts will install the service node, gateway and the redis
software itself on the same host. We can add more flexibility to this later.
2. For now, we use roles to enforce chef recipe dependencies. It seems to be
working well for what we have right now and I like the fact that there is one
location that maintains the chef dependencies for a given component.
3. For now, not all configurations of multi host are tested. In fact, I have
only
verified NATS. The config file template changes for all the other components
will be added in later changes. e.g. with this change you cannot run ccdb on a
separate host and expect cloud controller to work. But the changes to make it
work are more about adding the right templated fields and testing those changes.
Testing Done: I have verified that I can install "nats" on one box and the rest
of the components on a different box and things work fine.
Change-Id: I165b01fd65e4283748cf2cf9b2438369ae6332ce
Added template files for all cf components including services.
Add comments to the deployment yaml config file. Cleanup some of the scripts.
Change-Id: I9209749ab9ca50a2bd894189c571e1b4c33bc77b
Remove the stock recipes downloaded from opscode website. These were all
written from scratch.
Testing Done: BVT tests
Change-Id: Ibf2cc130e1c80941173ceba3dd4d5574b20d7c14
This is part 1 of what I think will be a 3 part change. This change adds the
basic chef recipes to deploy cloudfoundry on a single host. The major change is
to install ruby1.8 and ruby1.9 into a configurable directory. As a result I
added a wrapper script to invoke vcap start that sets up the ruby paths
appropriately before starting indivdual components. Note, these changes do not
download the vcap repo (I will add that in a later change).
Most of the chef recipes were downloaded from the opscode website. Some of them
have been modified to meet our requirements.
Usage:
1. run dev_setup/bin/vcap_dev_setup to install all the components required
by cloud foundry
2. run dev_setup/bin/vcap_dev {start, stop, restart} to start cloud foundry
components.
Testing: Installed a base Ubuntu image, ran bvts on cloud foundry installed
using these new scripts.
Change-Id: I22cf25131ae8f8ab725408521a44401578eceedd
Dr Nic Williams
Mahesh
anferneeg
AB Srinivasan
Katsunori Kawaguchi
Alex Pop
mpage
Albert Choi
Tang Rui
Alex Suraci
Patrick Bozeman
Nicholas Kushmerick
Woon Jung
Patrick Armstrong